Nov 01 20:27:55.062959 ip-10-0-131-188 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 20:27:55.062973 ip-10-0-131-188 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 20:27:55.062983 ip-10-0-131-188 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 20:27:55.063311 ip-10-0-131-188 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 20:28:06.490924 ip-10-0-131-188 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 20:28:06.490939 ip-10-0-131-188 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 142d24daa6ed4812bce3794c5ad81ecf --
Nov 01 20:30:00.600727 ip-10-0-131-188 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 20:30:00.967383 ip-10-0-131-188 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:30:00.967383 ip-10-0-131-188 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 20:30:00.967383 ip-10-0-131-188 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:30:00.967383 ip-10-0-131-188 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:30:00.972635 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972564    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.972635 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972634    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.972635 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972638    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972641    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972645    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972648    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972651    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972655    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972658    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972661    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972667    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972669    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972672    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972675    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972677    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972680    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972683    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972685    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972687    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972691    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972696    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972699    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.972723 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972701    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972704    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972706    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972709    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972711    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972713    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972716    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972718    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972721    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972723    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972726    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972729    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972731    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972734    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972737    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972739    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972742    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972744    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972747    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.973175 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972749    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972752    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972755    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972757    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972760    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972762    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972765    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972767    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972769    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972772    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972774    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972777    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972779    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972782    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972785    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972788    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972790    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972793    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972796    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972798    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.973612 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972801    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972803    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972806    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972808    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972811    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972813    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972816    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972818    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972820    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972823    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972825    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972828    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972830    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972833    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972836    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972838    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972841    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972843    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972846    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.974086 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972848    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972850    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972853    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972856    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972860    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972863    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972866    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972869    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972872    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972875    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972877    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972880    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972882    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972885    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972887    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972890    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972893    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972895    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972897    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.974547 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972900    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972903    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972905    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972908    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972910    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972913    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.972922    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973699    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973715    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973718    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973720    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973723    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973726    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973728    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973731    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973733    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973737    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973740    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973743    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.974998 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973745    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973749    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973752    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973754    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973757    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973759    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973762    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973764    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973767    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973769    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973773    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973777    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973780    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973783    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973789    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973791    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973794    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973796    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973799    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.975515 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973801    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973804    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973810    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973813    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973816    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973818    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973821    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973823    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973826    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973829    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973831    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973834    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973837    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973840    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973842    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973845    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973847    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973850    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973852    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.975974 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973855    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973857    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973860    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973862    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973865    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973867    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973870    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973872    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973878    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973881    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973883    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973886    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973889    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973891    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973894    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973896    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973899    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973902    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973904    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.976459 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973906    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973909    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973912    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973914    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973916    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973920    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973923    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973926    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973930    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973935    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973939    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973943    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973947    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973951    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973954    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973958    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973961    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973964    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973967    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973969    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.976984 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973972    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973974    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973978    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973982    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973986    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973989    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973992    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973994    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973997    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.973999    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974005    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974009    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974012    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974014    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974017    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974548    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974555    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974558    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974562    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.977458 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974565    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974567    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974570    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974573    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974576    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974579    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974582    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974584    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974587    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974589    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974592    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974595    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974597    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974600    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974603    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974605    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974608    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974610    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974613    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.977893 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974615    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974618    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974620    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974623    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974626    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974629    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974631    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974634    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974636    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974639    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974641    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974644    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974648    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974652    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974655    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974657    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974660    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974664    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974667    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974670    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.978416 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974672    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974675    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974677    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974680    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974682    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974685    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974687    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974690    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974693    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974695    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974698    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974700    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974703    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974705    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974708    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974710    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974713    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974716    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974719    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.978880 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974721    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974724    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974726    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974729    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974731    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974734    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974736    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974739    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974741    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974744    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974746    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974749    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974751    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974754    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974756    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974759    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974761    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974764    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974767    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974769    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.979320 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974772    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974775    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974777    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974780    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974782    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974785    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974787    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974789    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974792    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974794    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974797    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974800    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974803    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974805    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974808    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974810    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974813    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974815    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974818    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.979781 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974822    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974826    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974829    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974832    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.974834    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975842    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975852    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975857    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975862    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975866    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975869    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975873    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975877    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975880    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975883    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975887    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975890    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975893    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975896    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975899    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975903    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975906    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975909    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 20:30:00.980216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975914    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975917    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975921    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975924    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975927    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975931    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975934    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975937    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975940    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975943    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975946    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975949    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975952    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975955    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975958    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975961    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975964    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975967    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975970    2577 flags.go:64] FLAG: --enable-server="true"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975973    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975977    2577 flags.go:64] FLAG: --event-burst="100"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975980    2577 flags.go:64] FLAG: --event-qps="50"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975983    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975987    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 20:30:00.980763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975990    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975994    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.975997    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976000    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976003    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976005    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976008    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976011    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976014    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976017    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976020    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976023    2577 flags.go:64] FLAG: --feature-gates=""
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976027    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976030    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976034    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976036    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976039    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976043    2577 flags.go:64] FLAG: --help="false"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976046    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976050    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976053    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976056    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976059    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 20:30:00.981357 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976062    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976066    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976068    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976071    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976074    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976077    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976080    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976083    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976086    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976089    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976092    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976095    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976097    2577 flags.go:64] FLAG: --lock-file=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976100    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976103    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976106    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976111    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976114    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976117    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976120    2577 flags.go:64] FLAG: --logging-format="text"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976123    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976127    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976129    2577 flags.go:64] FLAG: --manifest-url=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976132    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976137    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 20:30:00.981910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976140    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976143    2577 flags.go:64] FLAG: --max-pods="110"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976146    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976149    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976152    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976154    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976157    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976160    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976163    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976172    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976175    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976178    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976181    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976184    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976189    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976192    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976194    2577 flags.go:64] FLAG: --port="10250"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976198    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976200    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a2325141049c29c0"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976204    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976207    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976209    2577 flags.go:64] FLAG: --register-node="true"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976212    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976216    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976219    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976221    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 20:30:00.982517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976224    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976228    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976231    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976234    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976237    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976240    2577 flags.go:64] FLAG: --runonce="false"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976243    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976246    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976249    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976252    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976254    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976257    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976260    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976263    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976266    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976269    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976272    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976274    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976277    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976280    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976284    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976290    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976293    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976295    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976299    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 01 20:30:00.983434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976302    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976305    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976308    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976311    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976315    2577 flags.go:64] FLAG: --v="2"
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976319    2577 flags.go:64] FLAG: --version="false"
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976323    2577 flags.go:64] FLAG: --vmodule=""
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976327    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976331    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976465    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976469    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976473    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976476    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976478    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976481    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976483    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976486    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976489    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976491    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976494    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976511    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.984088 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976515    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976519    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976522    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976524    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976527    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976530    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976534    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976537    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976540    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976542    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976545    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976548    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976550    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976553    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976556    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976560    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976562    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976565    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976568    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976570    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.984593 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976573    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976576    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976578    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976581    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976584    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976587    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976589    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976592    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976595    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976597    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976602    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976604    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976607    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976609    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976612    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976614    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976617    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976621    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976624    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.985061 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976626    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976629    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976632    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976634    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976637    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976640    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976643    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976648    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976651    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976654    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976657    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976660    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976662    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976665    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976668    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976670    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976674    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976676    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976679    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976681    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.985572 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976684    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976688    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976691    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976694    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976697    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976700    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976702    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976705    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976707    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976710    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976714    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976717    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976719    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976722    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976724    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976727    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976730    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976732    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976735    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.986028 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976738    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976741    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976744    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976746    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976752    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976754    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976757    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976760    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976762    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976765    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976768    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976771    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976773    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976776    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976778    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.976804    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:30:00.986489 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976838    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976842    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976845    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976849    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976852    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976855    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976857    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976861    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976864    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976866    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976869    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976872    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976874    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976877    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976879    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976882    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976884    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976888    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976891    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.986855 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976894    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976896    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976899    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976901    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976904    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976907    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976909    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976912    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976921    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976926    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976928    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976931    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976933    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976936    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976939    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976941    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976944    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976946    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976949    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976952    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.987281 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976956    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976959    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976961    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976964    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976966    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976969    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976971    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976974    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976977    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976979    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976983    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976985    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976988    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976991    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976994    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976996    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.976999    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977002    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977004    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977007    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.987751 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977010    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977013    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977015    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977018    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977021    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977023    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977026    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977028    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977031    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977033    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977036    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977039    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977042    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977045    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977049    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977053    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977055    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977058    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977060    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977063    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.988194 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977065    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977068    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977071    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977074    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977077    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977079    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977082    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977084    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977087    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977091    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977095    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977097    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977100    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977103    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977106    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977108    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977111    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977114    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977117    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.988770 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977119    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977122    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977124    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977127    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977129    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977133    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977135    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.977138    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.984767    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.984781    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.984798    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.984803    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984869    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984874    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984877    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984880    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984884    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984887    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984890    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.989210 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984892    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984897    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984900    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984904    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984907    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984910    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984912    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984923    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984927    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984931    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984934    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984937    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984939    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984942    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984945    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984947    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984950    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984952    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984959    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.989664 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984963    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984965    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984968    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984971    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984974    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984977    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984980    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984982    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984985    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984988    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984990    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984993    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984995    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.984998    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985000    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985003    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985006    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985009    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985012    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985014    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.990090 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985017    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985020    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985023    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985025    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985028    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985030    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985033    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985035    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985038    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985041    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985044    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985046    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985049    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985051    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985054    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985057    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985060    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985062    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985066    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985068    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.990600 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985071    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985074    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985076    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985079    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985082    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985084    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985087    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985089    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985092    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985095    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985097    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985100    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985102    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985106    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985109    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985113    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985116    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985118    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985121    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.991055 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985123    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985126    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985129    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985132    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985134    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985137    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985139    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985142    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985145    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985148    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985150    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985153    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985155    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985158    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985160    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985164    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985168    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985171    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985173    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.991524 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985176    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.985202    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985355    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985360    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985363    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985366    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985369    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985372    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985375    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985378    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985380    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985383    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985387    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985403    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985407    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985410    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.991956 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985414    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985416    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985419    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985422    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985424    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985427    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985430    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985433    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985435    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985438    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985441    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985443    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985446    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985448    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985451    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985453    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985456    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985458    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985461    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.992311 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985463    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985466    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985468    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985471    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985474    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985477    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985479    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985482    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985485    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985487    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985490    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985493    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985495    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985498    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985500    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985503    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985505    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985508    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985510    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.992792 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985513    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985515    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985518    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985521    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985523    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985526    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985528    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985531    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985533    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985536    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985538    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985541    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985543    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985546    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985549    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985552    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985556    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985558    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985561    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985565    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.993220 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985567    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985570    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985573    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985575    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985578    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985581    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985584    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985586    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985589    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985592    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985594    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985597    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985600    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985604    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985607    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985609    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985612    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985614    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985617    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.993688 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985619    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985622    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985624    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985627    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985629    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985632    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985634    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985637    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985639    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985642    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985649    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985651    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985654    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985657    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.985679    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985710    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:00.994131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985714    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985717    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985721    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985724    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985727    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985729    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985732    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985735    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985737    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985740    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985743    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985746    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985748    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985751    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985753    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985756    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985759    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985762    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985765    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985768    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:00.994504 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985770    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985773    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985776    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985778    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985780    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985783    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985786    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985788    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985791    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985794    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985796    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985799    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985802    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985804    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985806    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985809    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985811    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985814    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985816    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985819    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:00.994968 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985822    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985824    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985827    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985829    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985832    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985834    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985837    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985839    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985841    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985844    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985846    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985849    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985851    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985854    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985856    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985858    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985861    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985863    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985866    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:00.995424 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985871    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985874    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985877    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985880    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985882    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985886    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985890    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985893    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985895    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985898    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985901    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985903    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985906    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985908    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985911    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985913    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985916    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985918    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985921    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:00.995903 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985923    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985926    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985928    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985931    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985933    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985935    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985938    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985940    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985943    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985945    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985948    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985950    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985953    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985955    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985959    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985961    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985964    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985967    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985970    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:00.996344 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985972    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985974    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985979    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985982    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985985    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:00.985987    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.986032    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.988318    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.989298    2577 server.go:1014] "Starting client certificate rotation"
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.989402    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.989444    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 20:30:00.996846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.992237    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:30:00.997142 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:00.996999    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:30:01.010441 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.010427    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 01 20:30:01.014120 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.014107    2577 log.go:25] "Validated CRI v1 image API"
Nov 01 20:30:01.014934 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.014913    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 20:30:01.019781 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.019765    2577 fs.go:135] Filesystem UUIDs: map[4cbcd29a-6d9f-429c-b003-a24b39241eee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d91d2445-defb-4f2c-8605-5b36e046690e:/dev/nvme0n1p3]
Nov 01 20:30:01.019842 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.019782    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 20:30:01.025172 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.025066    2577 manager.go:217] Machine: {Timestamp:2026-11-01 20:30:01.023423041 +0000 UTC m=+0.330072529 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3205875 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24ce3f1506e987e9029f3a8eb3a53d SystemUUID:ec24ce3f-1506-e987-e902-9f3a8eb3a53d BootID:142d24da-a6ed-4812-bce3-794c5ad81ecf Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:33:7c:81:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:33:7c:81:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:4c:3b:71:c5:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 20:30:01.025172 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.025163    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 20:30:01.025314 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.025232    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 20:30:01.028756 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.028732    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 20:30:01.028903 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.028758    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-188.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 20:30:01.028993 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.028907    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 20:30:01.028993 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.028925    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 20:30:01.028993 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.028956    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:30:01.029837 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.029826    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:30:01.031057 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.031045    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 20:30:01.035645 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.035632    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 20:30:01.035778 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.035767    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 20:30:01.036612 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.036602    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 20:30:01.036672 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.036619    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 20:30:01.036672 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.036632    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 20:30:01.036672 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.036645    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 01 20:30:01.036672 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.036656    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 20:30:01.036977 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.036964    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:30:01.039289 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.039273    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 20:30:01.045910 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.045897    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:30:01.046779 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046767    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 20:30:01.046814 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046798    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 20:30:01.046953 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046943    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 20:30:01.046984 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046959    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 20:30:01.046984 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046965    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 20:30:01.046984 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046970    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 20:30:01.046984 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046975    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 20:30:01.046984 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.046982    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 20:30:01.047592 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.047584    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 20:30:01.047628 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.047593    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 20:30:01.047628 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.047600    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 20:30:01.047628 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.047612    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 20:30:01.047628 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.047624    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 20:30:01.047819 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.047633    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 20:30:01.049436 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.049426    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 20:30:01.049475 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.049439    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 20:30:01.053175 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.053164    2577 server.go:1263] "Started kubelet"
Nov 01 20:30:01.053309 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.053274    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 20:30:01.053309 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.053278    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 20:30:01.053421 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.053341    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 20:30:01.053859 ip-10-0-131-188 systemd[1]: Started Kubernetes Kubelet.
Nov 01 20:30:01.056049 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.056033    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 20:30:01.056233 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.056221    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 20:30:01.061556 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.061538    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 20:30:01.062489 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.062475    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 20:30:01.063036 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063023    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 20:30:01.063156 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.063132    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-188.us-west-1.compute.internal\" not found"
Nov 01 20:30:01.063770 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063751    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 20:30:01.063770 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063759    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 20:30:01.063894 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063778    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 20:30:01.063894 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063859    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 20:30:01.063894 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063866    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 20:30:01.063894 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063877    2577 factory.go:55] Registering systemd factory
Nov 01 20:30:01.063894 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063883    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 01 20:30:01.064103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.063868    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 20:30:01.064103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.064058    2577 factory.go:153] Registering CRI-O factory
Nov 01 20:30:01.064103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.064065    2577 factory.go:223] Registration of the crio container factory successfully
Nov 01 20:30:01.064103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.064076    2577 factory.go:103] Registering Raw factory
Nov 01 20:30:01.064103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.064087    2577 manager.go:1199] Started watching for new ooms in manager
Nov 01 20:30:01.065114 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.065099    2577 manager.go:319] Starting recovery of all containers
Nov 01 20:30:01.075386 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.075370    2577 manager.go:324] Recovery completed
Nov 01 20:30:01.081795 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.081782    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.083281 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083265    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.083325 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083294    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.083325 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083304    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.083797 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083784    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 01 20:30:01.083866 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083797    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 20:30:01.083866 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083786    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.083866 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.083817    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 20:30:01.084481 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.084458    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.084541 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.084486    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.084541 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.084499    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.086027 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.086015    2577 policy_none.go:50] "Start"
Nov 01 20:30:01.086027 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.086029    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 20:30:01.086101 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.086042    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 20:30:01.087991 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.087982    2577 policy_none.go:44] "Start"
Nov 01 20:30:01.112451 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.112420    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 20:30:01.114690 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.114677    2577 manager.go:341] "Starting Device Plugin manager"
Nov 01 20:30:01.114762 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.114730    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 20:30:01.114762 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.114741    2577 server.go:87] "Starting device plugin registration server"
Nov 01 20:30:01.114937 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.114925    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 20:30:01.114999 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.114938    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 20:30:01.115053 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.115021    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 20:30:01.115469 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.115103    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 20:30:01.115469 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.115116    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 20:30:01.115590 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.115543    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 20:30:01.115663 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.115645    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-188.us-west-1.compute.internal\" not found"
Nov 01 20:30:01.195098 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.193980    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-188.us-west-1.compute.internal.18e405dda1586e0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-188.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-188.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:01.053138446 +0000 UTC m=+0.359787932,LastTimestamp:2026-11-01 20:30:01.053138446 +0000 UTC m=+0.359787932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-188.us-west-1.compute.internal,}"
Nov 01 20:30:01.215192 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.215172    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.215898 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.215883    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.215956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.215908    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.215956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.215929    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.215956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.215955    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.242287 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.242234    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-188.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 20:30:01.242679 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.242657    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-188.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 20:30:01.262356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.262336    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 20:30:01.262417 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.262410    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x4qq4"
Nov 01 20:30:01.308584 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.308562    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 20:30:01.310559 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.309656    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 20:30:01.310559 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.309680    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 20:30:01.310559 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.309700    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 20:30:01.310559 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.309735    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 20:30:01.348588 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.348571    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:01.349960 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.349947    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:01.410311 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.410288    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"]
Nov 01 20:30:01.410385 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.410378    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.411149 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.411129    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.411227 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.411161    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.411227 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.411177    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.413495 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.413483    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.413640 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.413622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.413676 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.413654    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.414214 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.414197    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.414214 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.414208    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.414334 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.414226    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.414334 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.414231    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.414334 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.414242    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.414334 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.414245    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.416359 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.416346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.416421 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.416371    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:01.417147 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.417127    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:01.417230 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.417151    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:01.417230 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.417161    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:01.439848 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.439827    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-188.us-west-1.compute.internal\" not found" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.442622 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.442607    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-188.us-west-1.compute.internal\" not found" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.449666 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.449644    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.449758 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.449672    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-188.us-west-1.compute.internal\": node \"ip-10-0-131-188.us-west-1.compute.internal\" not found"
Nov 01 20:30:01.449999 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.449983    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:01.451283 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.451267    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:01.463290 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.463275    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.465637 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.465622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/434dabbcb9fcd49f00b858143ea6ae44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"434dabbcb9fcd49f00b858143ea6ae44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.465723 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.465645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.465723 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.465662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.566345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.566306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.566345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.566328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.566345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.566344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/434dabbcb9fcd49f00b858143ea6ae44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"434dabbcb9fcd49f00b858143ea6ae44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.566509 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.566387    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/434dabbcb9fcd49f00b858143ea6ae44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"434dabbcb9fcd49f00b858143ea6ae44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.566509 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.566422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.566509 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.566422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.572554 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.572541    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:30:01.573181 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.573170    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.629052 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.629037    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:30:01.743482 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.743464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.746601 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.746589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:01.989323 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.989266    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 20:30:01.989763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.989365    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 20:30:01.989763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.989412    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 20:30:01.989763 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:01.989420    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 20:30:01.989763 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:01.989419    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9584bc60fd4645d0a6dd8d98805fda7-6263b06240bad7c4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-188.us-west-1.compute.internal.18e405dda324c88b\": read tcp 10.0.131.188:48402->18.224.125.164:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-188.us-west-1.compute.internal.18e405dda324c88b  default   4815 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-188.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-188.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-188.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:01 +0000 UTC,LastTimestamp:2026-11-01 20:30:01.215932599 +0000 UTC m=+0.522582086,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-188.us-west-1.compute.internal,}"
Nov 01 20:30:02.037158 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:02.037139    2577 apiserver.go:52] "Watching apiserver"
Nov 01 20:30:02.063148 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:02.063127    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 20:30:03.395457 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.395380    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 20:25:02 +0000 UTC" deadline="2028-06-30 21:35:42.475178254 +0000 UTC"
Nov 01 20:30:03.395457 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.395420    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14569h5m39.079761336s"
Nov 01 20:30:03.428188 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.428167    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 20:30:03.428484 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.428469    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-f7pr8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-fspt2","openshift-multus/network-metrics-daemon-cx7ds","openshift-dns/node-resolver-92gcr","openshift-network-operator/iptables-alerter-l7c7b","openshift-ovn-kubernetes/ovnkube-node-phs9n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq","openshift-cluster-node-tuning-operator/tuned-xbb76","openshift-image-registry/node-ca-w2766","kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal","openshift-multus/multus-t2v7x","openshift-network-diagnostics/network-check-target-j4t6k","kube-system/global-pull-secret-syncer-8x2dv"]
Nov 01 20:30:03.430894 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.430879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.432951 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.432935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.434932 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.434918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:03.434999 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.434969    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:03.436962 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.436943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.439048 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.439029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.441266 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.441249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.443505 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.443488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.445791 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.445775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.447963 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.447949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.450184 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.450171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.452288 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.452273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:03.452354 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.452321    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:03.454381 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.454367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.454453 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.454413    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:03.483639 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.483617    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 20:30:03.483810 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.483795    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 20:30:03.483881 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.483813    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r4cnd\""
Nov 01 20:30:03.484939 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.484926    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.485323 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.485306    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j72sw\""
Nov 01 20:30:03.485563 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.485552    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 20:30:03.485603 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.485563    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.485687 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.485677    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 20:30:03.485724 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.485691    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 20:30:03.489312 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.489298    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pjdf7\""
Nov 01 20:30:03.489490 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.489474    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.489490 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.489475    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.493129 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.493114    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 20:30:03.494813 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.494798    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5cgzw\""
Nov 01 20:30:03.494883 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.494856    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.494883 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.494866    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 20:30:03.494883 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.494871    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.494883 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.494873    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wk8gn\""
Nov 01 20:30:03.494996 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.494899    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.495386 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.495375    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 20:30:03.495421 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.495397    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 20:30:03.495774 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.495763    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.495828 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.495816    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 20:30:03.499126 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-host\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.499217 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.499217 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-k8s-cni-cncf-io\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.499217 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cnibin\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.499217 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfzs\" (UniqueName: \"kubernetes.io/projected/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-kube-api-access-frfzs\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.499360 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-script-lib\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.499360 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.499360 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-host\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.499360 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32264030-2487-4a8d-b5e4-3f233831760b-konnectivity-ca\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.499360 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.499360 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-netns\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6559ca8c-d167-4f33-95f9-1856858fc3d5-hosts-file\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-system-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9nk\" (UniqueName: \"kubernetes.io/projected/5318ea66-6db7-4701-a20d-84b25e4c196c-kube-api-access-zv9nk\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-tuned\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-bin\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5318ea66-6db7-4701-a20d-84b25e4c196c-host-slash\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.499568 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-run\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-sys\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-serviceca\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-kubelet\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6559ca8c-d167-4f33-95f9-1856858fc3d5-tmp-dir\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-multus\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdqs\" (UniqueName: \"kubernetes.io/projected/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-kube-api-access-zgdqs\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32264030-2487-4a8d-b5e4-3f233831760b-agent-certs\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.499854 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-conf-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-etc-kubernetes\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-multus-daemon-config\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp849\" (UniqueName: \"kubernetes.io/projected/26fed350-454f-4f57-8989-016d524f5b4f-kube-api-access-lp849\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5277\" (UniqueName: \"kubernetes.io/projected/6559ca8c-d167-4f33-95f9-1856858fc3d5-kube-api-access-b5277\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.499990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8wh\" (UniqueName: \"kubernetes.io/projected/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-kube-api-access-4x8wh\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-socket-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-kubernetes\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxpg\" (UniqueName: \"kubernetes.io/projected/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-kube-api-access-zkxpg\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:03.500100 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-env-overrides\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-registration-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysconfig\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-conf\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-var-lib-kubelet\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-netns\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-socket-dir-parent\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lcr\" (UniqueName: \"kubernetes.io/projected/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kube-api-access-q4lcr\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-modprobe-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-kubelet\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-var-lib-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-hostroot\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-systemd-units\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-systemd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-systemd\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-slash\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-etc-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-ovn\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-log-socket\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-config\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktzx\" (UniqueName: \"kubernetes.io/projected/428f74c3-acde-4cbb-afd8-32c13b30cadf-kube-api-access-wktzx\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-netd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-cni-binary-copy\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-bin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.500742 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-multus-certs\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-kubelet-config\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-dbus\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-tmp\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-node-log\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-system-cni-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-sys-fs\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-cnibin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5318ea66-6db7-4701-a20d-84b25e4c196c-iptables-alerter-script\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-device-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.501146 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.500944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-lib-modules\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.531955 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.531939    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 20:30:03.534144 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534123    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9xjd\""
Nov 01 20:30:03.534144 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534137    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 20:30:03.534244 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534154    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.534416 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534403    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bdld\""
Nov 01 20:30:03.534480 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534433    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.534480 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534439    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.534577 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534480    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.534577 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534495    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 20:30:03.534577 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534504    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ggf7k\""
Nov 01 20:30:03.534577 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534509    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h288r\""
Nov 01 20:30:03.534577 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534527    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 20:30:03.534577 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534539    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 20:30:03.534851 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534598    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 20:30:03.534851 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534730    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 20:30:03.534851 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.534755    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jmrrp"
Nov 01 20:30:03.601754 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-netns\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.601754 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6559ca8c-d167-4f33-95f9-1856858fc3d5-hosts-file\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-system-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9nk\" (UniqueName: \"kubernetes.io/projected/5318ea66-6db7-4701-a20d-84b25e4c196c-kube-api-access-zv9nk\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-netns\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-tuned\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6559ca8c-d167-4f33-95f9-1856858fc3d5-hosts-file\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-system-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-bin\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.601890 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5318ea66-6db7-4701-a20d-84b25e4c196c-host-slash\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-run\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-sys\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5318ea66-6db7-4701-a20d-84b25e4c196c-host-slash\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-serviceca\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-bin\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.601998    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-sys\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-kubelet\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602043    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6559ca8c-d167-4f33-95f9-1856858fc3d5-tmp-dir\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602061    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-kubelet\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-multus\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-multus\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdqs\" (UniqueName: \"kubernetes.io/projected/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-kube-api-access-zgdqs\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.602261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602133    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602364    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6559ca8c-d167-4f33-95f9-1856858fc3d5-tmp-dir\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602391    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-serviceca\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602506    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-run\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32264030-2487-4a8d-b5e4-3f233831760b-agent-certs\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-conf-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-etc-kubernetes\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-multus-daemon-config\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp849\" (UniqueName: \"kubernetes.io/projected/26fed350-454f-4f57-8989-016d524f5b4f-kube-api-access-lp849\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602704    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-conf-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-etc-kubernetes\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5277\" (UniqueName: \"kubernetes.io/projected/6559ca8c-d167-4f33-95f9-1856858fc3d5-kube-api-access-b5277\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8wh\" (UniqueName: \"kubernetes.io/projected/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-kube-api-access-4x8wh\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-socket-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.602941 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-kubernetes\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-kubernetes\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602973    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-socket-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.602956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxpg\" (UniqueName: \"kubernetes.io/projected/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-kube-api-access-zkxpg\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-env-overrides\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-registration-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysconfig\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-conf\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-registration-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-var-lib-kubelet\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-multus-daemon-config\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603183    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-netns\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603193    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-var-lib-kubelet\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysconfig\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-netns\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603231    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-conf\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.603573 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-socket-dir-parent\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lcr\" (UniqueName: \"kubernetes.io/projected/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kube-api-access-q4lcr\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-socket-dir-parent\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-modprobe-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603361    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-modprobe-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-kubelet\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-var-lib-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603501    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-hostroot\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-hostroot\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-systemd-units\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-systemd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603953    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-systemd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-var-lib-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.603989    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-systemd-units\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.603456    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-kubelet\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.604280 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.604078    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:04.104048757 +0000 UTC m=+2.282436955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-systemd\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-slash\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-etc-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-ovn\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-log-socket\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-log-socket\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-config\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604357    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-slash\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604410    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-etc-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktzx\" (UniqueName: \"kubernetes.io/projected/428f74c3-acde-4cbb-afd8-32c13b30cadf-kube-api-access-wktzx\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-netd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.605003 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-ovn\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-cni-binary-copy\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-bin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604589    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604635    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-systemd\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.604722    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.604779    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret podName:1cdfd004-c495-4f69-9aba-35276b089b06 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:04.104762396 +0000 UTC m=+2.283150589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret") pod "global-pull-secret-syncer-8x2dv" (UID: "1cdfd004-c495-4f69-9aba-35276b089b06") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-multus-certs\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-kubelet-config\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-dbus\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-tmp\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-node-log\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.604984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-system-cni-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-sys-fs\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.610318 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-cnibin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5318ea66-6db7-4701-a20d-84b25e4c196c-iptables-alerter-script\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-device-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-lib-modules\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-host\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-k8s-cni-cncf-io\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cnibin\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfzs\" (UniqueName: \"kubernetes.io/projected/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-kube-api-access-frfzs\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-script-lib\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-host\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32264030-2487-4a8d-b5e4-3f233831760b-konnectivity-ca\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605569    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-kubelet-config\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605687    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-dbus\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:03.611182 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32264030-2487-4a8d-b5e4-3f233831760b-konnectivity-ca\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.605957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.606054    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.606523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.606900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-env-overrides\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.606955    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-system-cni-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.606992    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-node-log\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5318ea66-6db7-4701-a20d-84b25e4c196c-iptables-alerter-script\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607135    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-device-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-lib-modules\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607377    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-host\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-host\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607506    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-k8s-cni-cncf-io\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607563    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-bin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cnibin\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-sys-fs\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.611956 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607758    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-netd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607762    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-tuned\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607841    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-cnibin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32264030-2487-4a8d-b5e4-3f233831760b-agent-certs\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.607956    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-multus-certs\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.608622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-tmp\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.608638    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-cni-binary-copy\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.608760    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-script-lib\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.608868    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-config\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.612800 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.610157    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.660290 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.660240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5277\" (UniqueName: \"kubernetes.io/projected/6559ca8c-d167-4f33-95f9-1856858fc3d5-kube-api-access-b5277\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.660552 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.660525    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lcr\" (UniqueName: \"kubernetes.io/projected/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kube-api-access-q4lcr\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.660715 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.660691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxpg\" (UniqueName: \"kubernetes.io/projected/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-kube-api-access-zkxpg\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:03.661064 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.661048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9nk\" (UniqueName: \"kubernetes.io/projected/5318ea66-6db7-4701-a20d-84b25e4c196c-kube-api-access-zv9nk\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.661423 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.661395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8wh\" (UniqueName: \"kubernetes.io/projected/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-kube-api-access-4x8wh\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.661677 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.661643    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp849\" (UniqueName: \"kubernetes.io/projected/26fed350-454f-4f57-8989-016d524f5b4f-kube-api-access-lp849\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.661925 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.661911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdqs\" (UniqueName: \"kubernetes.io/projected/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-kube-api-access-zgdqs\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.662580 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.662564    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:30:03.662639 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.662587    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:30:03.662639 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.662600    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxxmw for pod openshift-network-diagnostics/network-check-target-j4t6k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:03.662758 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:03.662665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw podName:260814c3-535b-448a-9213-2b992657efb8 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:04.162636624 +0000 UTC m=+2.341024821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lxxmw" (UniqueName: "kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw") pod "network-check-target-j4t6k" (UID: "260814c3-535b-448a-9213-2b992657efb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:03.664404 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.664384    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfzs\" (UniqueName: \"kubernetes.io/projected/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-kube-api-access-frfzs\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.664495 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.664481    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktzx\" (UniqueName: \"kubernetes.io/projected/428f74c3-acde-4cbb-afd8-32c13b30cadf-kube-api-access-wktzx\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.758298 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.758272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:03.787718 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.787696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:03.795279 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.795263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:03.809691 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.809676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:03.836154 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.836137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:03.842576 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.842561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:03.852492 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.852473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:03.858320 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.858306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:03.862806 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:03.862787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:03.996113 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:03.996077    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905080b6348e87e21e0923f43ca1286f.slice/crio-56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977 WatchSource:0}: Error finding container 56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977: Status 404 returned error can't find the container with id 56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977
Nov 01 20:30:03.996487 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:03.996467    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434dabbcb9fcd49f00b858143ea6ae44.slice/crio-fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e WatchSource:0}: Error finding container fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e: Status 404 returned error can't find the container with id fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e
Nov 01 20:30:04.000417 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.000400    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:30:04.110491 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.110472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:04.110572 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.110507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:04.110621 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.110591    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:04.110621 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.110616    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:04.110707 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.110637    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:05.110622976 +0000 UTC m=+3.289011171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:04.110707 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.110650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret podName:1cdfd004-c495-4f69-9aba-35276b089b06 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:05.110643479 +0000 UTC m=+3.289031658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret") pod "global-pull-secret-syncer-8x2dv" (UID: "1cdfd004-c495-4f69-9aba-35276b089b06") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:04.184761 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.184742    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:04.210726 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.210711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:04.210805 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.210798    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:30:04.210843 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.210810    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:30:04.210843 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.210820    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxxmw for pod openshift-network-diagnostics/network-check-target-j4t6k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:04.210899 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:04.210862    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw podName:260814c3-535b-448a-9213-2b992657efb8 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:05.210851686 +0000 UTC m=+3.389239879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxxmw" (UniqueName: "kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw") pod "network-check-target-j4t6k" (UID: "260814c3-535b-448a-9213-2b992657efb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:04.235620 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.235603    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:04.410770 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.410728    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:04.441382 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.441347    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"434dabbcb9fcd49f00b858143ea6ae44","Type":"ContainerStarted","Data":"fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e"}
Nov 01 20:30:04.442249 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.442232    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"905080b6348e87e21e0923f43ca1286f","Type":"ContainerStarted","Data":"56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977"}
Nov 01 20:30:04.538015 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.537991    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:03 +0000 UTC" deadline="2028-06-19 16:34:05.853521574 +0000 UTC"
Nov 01 20:30:04.538015 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:04.538011    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14300h4m1.315513238s"
Nov 01 20:30:04.601822 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.601799    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32264030_2487_4a8d_b5e4_3f233831760b.slice/crio-0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86 WatchSource:0}: Error finding container 0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86: Status 404 returned error can't find the container with id 0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86
Nov 01 20:30:04.633993 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.633960    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ec5f83_b7c3_4e1b_aa70_5a3d3da96afd.slice/crio-84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919 WatchSource:0}: Error finding container 84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919: Status 404 returned error can't find the container with id 84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919
Nov 01 20:30:04.644430 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.644407    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6559ca8c_d167_4f33_95f9_1856858fc3d5.slice/crio-e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d WatchSource:0}: Error finding container e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d: Status 404 returned error can't find the container with id e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d
Nov 01 20:30:04.692538 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.692519    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce97c7cb_9ccb_468e_af69_a8f8e922ec24.slice/crio-7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609 WatchSource:0}: Error finding container 7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609: Status 404 returned error can't find the container with id 7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609
Nov 01 20:30:04.712890 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.712866    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3961b5_f2f0_4c47_9017_72de63a2f7c4.slice/crio-c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420 WatchSource:0}: Error finding container c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420: Status 404 returned error can't find the container with id c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420
Nov 01 20:30:04.735131 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.735113    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe4d6b3_46b7_4c61_88e0_a130c95e31d5.slice/crio-72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0 WatchSource:0}: Error finding container 72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0: Status 404 returned error can't find the container with id 72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0
Nov 01 20:30:04.754730 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.754709    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fed350_454f_4f57_8989_016d524f5b4f.slice/crio-96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88 WatchSource:0}: Error finding container 96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88: Status 404 returned error can't find the container with id 96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88
Nov 01 20:30:04.774890 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:04.774873    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428f74c3_acde_4cbb_afd8_32c13b30cadf.slice/crio-1f7e2678fbdf035afa3418495b270648f08601016eec981db5222c7a04eec8e0 WatchSource:0}: Error finding container 1f7e2678fbdf035afa3418495b270648f08601016eec981db5222c7a04eec8e0: Status 404 returned error can't find the container with id 1f7e2678fbdf035afa3418495b270648f08601016eec981db5222c7a04eec8e0
Nov 01 20:30:05.115896 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.115825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:05.115896 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.115863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:05.116093 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.115947    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:05.116093 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.115963    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:05.116093 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.116009    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:07.115993454 +0000 UTC m=+5.294381644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:05.116093 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.116027    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret podName:1cdfd004-c495-4f69-9aba-35276b089b06 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:07.11601906 +0000 UTC m=+5.294407242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret") pod "global-pull-secret-syncer-8x2dv" (UID: "1cdfd004-c495-4f69-9aba-35276b089b06") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:05.217118 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.217087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:05.217738 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.217209    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:30:05.217738 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.217222    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:30:05.217738 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.217232    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxxmw for pod openshift-network-diagnostics/network-check-target-j4t6k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:05.217738 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.217271    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw podName:260814c3-535b-448a-9213-2b992657efb8 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:07.217259129 +0000 UTC m=+5.395647308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxxmw" (UniqueName: "kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw") pod "network-check-target-j4t6k" (UID: "260814c3-535b-448a-9213-2b992657efb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:05.438505 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.438434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:05.438647 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.438548    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:05.438647 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.438624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:05.438779 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.438719    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:05.438779 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.438775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:05.438876 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:05.438836    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:05.446405 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.446376    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2v7x" event={"ID":"26fed350-454f-4f57-8989-016d524f5b4f","Type":"ContainerStarted","Data":"96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88"}
Nov 01 20:30:05.448049 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.448023    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w2766" event={"ID":"ce97c7cb-9ccb-468e-af69-a8f8e922ec24","Type":"ContainerStarted","Data":"7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609"}
Nov 01 20:30:05.450181 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.450156    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerStarted","Data":"72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0"}
Nov 01 20:30:05.451802 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.451775    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420"}
Nov 01 20:30:05.454508 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.454469    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l7c7b" event={"ID":"5318ea66-6db7-4701-a20d-84b25e4c196c","Type":"ContainerStarted","Data":"c04f0fc524901f8a7537b7c1c8c1bed1cd6f595100a32d48494684ecec0bd7cd"}
Nov 01 20:30:05.455692 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.455670    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92gcr" event={"ID":"6559ca8c-d167-4f33-95f9-1856858fc3d5","Type":"ContainerStarted","Data":"e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d"}
Nov 01 20:30:05.457288 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.457265    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq" event={"ID":"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd","Type":"ContainerStarted","Data":"84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919"}
Nov 01 20:30:05.458802 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.458779    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f7pr8" event={"ID":"32264030-2487-4a8d-b5e4-3f233831760b","Type":"ContainerStarted","Data":"0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86"}
Nov 01 20:30:05.460174 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.460149    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xbb76" event={"ID":"428f74c3-acde-4cbb-afd8-32c13b30cadf","Type":"ContainerStarted","Data":"1f7e2678fbdf035afa3418495b270648f08601016eec981db5222c7a04eec8e0"}
Nov 01 20:30:05.539094 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.539047    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:03 +0000 UTC" deadline="2028-04-19 20:17:34.172880697 +0000 UTC"
Nov 01 20:30:05.539272 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:05.539089    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12839h47m28.633794742s"
Nov 01 20:30:07.132736 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:07.132649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:07.132736 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:07.132715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:07.133251 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.132851    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:07.133251 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.132906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret podName:1cdfd004-c495-4f69-9aba-35276b089b06 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.132890098 +0000 UTC m=+9.311278284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret") pod "global-pull-secret-syncer-8x2dv" (UID: "1cdfd004-c495-4f69-9aba-35276b089b06") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:07.133362 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.133283    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:07.133362 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.133329    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.133314323 +0000 UTC m=+9.311702505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:07.233667 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:07.233590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:07.233830 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.233741    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:30:07.233830 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.233764    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:30:07.233830 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.233778    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxxmw for pod openshift-network-diagnostics/network-check-target-j4t6k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:07.233830 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.233825    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw podName:260814c3-535b-448a-9213-2b992657efb8 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.23380926 +0000 UTC m=+9.412197446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxxmw" (UniqueName: "kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw") pod "network-check-target-j4t6k" (UID: "260814c3-535b-448a-9213-2b992657efb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:07.439599 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:07.439159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:07.439599 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:07.439194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:07.439599 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:07.439165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:07.439599 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.439290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:07.439599 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.439382    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:07.439599 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:07.439453    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:09.439321 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.438718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:09.439321 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.438929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:09.439321 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:09.439043    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:09.439321 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:09.439145    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:09.439321 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.439202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:09.439321 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:09.439273    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:09.497703 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.497187    2577 generic.go:356] "Generic (PLEG): container finished" podID="905080b6348e87e21e0923f43ca1286f" containerID="a89aeefefd04a0b64514117dc3a5b065ca0b0e32d798602f21f31f345ae42141" exitCode=0
Nov 01 20:30:09.497703 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.497266    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"905080b6348e87e21e0923f43ca1286f","Type":"ContainerDied","Data":"a89aeefefd04a0b64514117dc3a5b065ca0b0e32d798602f21f31f345ae42141"}
Nov 01 20:30:09.500162 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.500133    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="01c9d88cbb0bb83f1f29b43fbef4edfe531b5f4807089608dc7ced9476b62700" exitCode=0
Nov 01 20:30:09.500267 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.500220    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerDied","Data":"01c9d88cbb0bb83f1f29b43fbef4edfe531b5f4807089608dc7ced9476b62700"}
Nov 01 20:30:09.507468 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.507400    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"434dabbcb9fcd49f00b858143ea6ae44","Type":"ContainerStarted","Data":"b6ab0c2705e17a836ebb3c38c826150ef861ef317fbf84c817ca271e7bf74a87"}
Nov 01 20:30:09.847884 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:09.847818    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal" podStartSLOduration=7.84780053 podStartE2EDuration="7.84780053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:09.729248448 +0000 UTC m=+7.907636674" watchObservedRunningTime="2026-11-01 20:30:09.84780053 +0000 UTC m=+8.026188731"
Nov 01 20:30:10.512925 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:10.512852    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"905080b6348e87e21e0923f43ca1286f","Type":"ContainerStarted","Data":"ce1338c36a2a5f12d83fa771fecf576ef5d1618c79b4cfa16c5669799688b3a8"}
Nov 01 20:30:10.519028 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:10.518997    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f7pr8" event={"ID":"32264030-2487-4a8d-b5e4-3f233831760b","Type":"ContainerStarted","Data":"f1d8a93c00a827278e042136bafa4a6df421cb54df42f708dfca9b726cb5883a"}
Nov 01 20:30:10.525835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:10.525807    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w2766" event={"ID":"ce97c7cb-9ccb-468e-af69-a8f8e922ec24","Type":"ContainerStarted","Data":"450d17732ae9aa14c580be621c409f855f8601690a978e2472d7c4f9e981c1c8"}
Nov 01 20:30:10.741045 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:10.740980    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" podStartSLOduration=8.740960448 podStartE2EDuration="8.740960448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:10.627572569 +0000 UTC m=+8.805960773" watchObservedRunningTime="2026-11-01 20:30:10.740960448 +0000 UTC m=+8.919348648"
Nov 01 20:30:10.856495 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:10.856452    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w2766" podStartSLOduration=3.314035795 podStartE2EDuration="8.856434512s" totalImagesPullingTime="5.542398717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:10.742165832 +0000 UTC m=+8.920554033" watchObservedRunningTime="2026-11-01 20:30:10.856434512 +0000 UTC m=+9.034822714"
Nov 01 20:30:11.163874 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.163799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:11.163874 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.163848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:11.164064 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.164009    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:11.164112 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.164065    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret podName:1cdfd004-c495-4f69-9aba-35276b089b06 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:19.164048133 +0000 UTC m=+17.342436318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret") pod "global-pull-secret-syncer-8x2dv" (UID: "1cdfd004-c495-4f69-9aba-35276b089b06") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:11.164184 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.164128    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:11.164184 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.164160    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:19.164149425 +0000 UTC m=+17.342537605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:11.264497 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.264464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:11.264693 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.264600    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:30:11.264693 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.264620    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:30:11.264693 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.264632    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxxmw for pod openshift-network-diagnostics/network-check-target-j4t6k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:11.264868 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.264699    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw podName:260814c3-535b-448a-9213-2b992657efb8 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:19.264681274 +0000 UTC m=+17.443069460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxxmw" (UniqueName: "kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw") pod "network-check-target-j4t6k" (UID: "260814c3-535b-448a-9213-2b992657efb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:11.439156 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.439083    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:11.439324 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.439215    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:11.439590 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.439575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:11.439700 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.439678    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:11.439773 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.439749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:11.439846 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:11.439829    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:11.531400 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:11.531371    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq" event={"ID":"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd","Type":"ContainerStarted","Data":"52be18e55b8c63e35211762bb16aee98bc8514d8bfa6b2f54eedc52768bf9f0d"}
Nov 01 20:30:13.027300 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.027073    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 20:30:13.249011 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.248853    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:30:13.027276396Z","UUID":"032eae31-ce56-4bf6-9b1c-d5ecaa59470d","Handler":null,"Name":"","Endpoint":""}
Nov 01 20:30:13.250730 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.250371    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 20:30:13.250730 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.250405    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 20:30:13.438473 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.438441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:13.438628 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.438453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:13.438628 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.438559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:13.438628 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:13.438560    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:13.438836 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:13.438676    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:13.438836 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:13.438743    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:13.525314 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.525247    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:13.525948 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.525929    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:13.537708 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.537682    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq" event={"ID":"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd","Type":"ContainerStarted","Data":"a0df9ce09f4a16648a3f94da8a5998bd598b61eeb709b47f917d8c356256a91a"}
Nov 01 20:30:13.539601 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.539577    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2v7x" event={"ID":"26fed350-454f-4f57-8989-016d524f5b4f","Type":"ContainerStarted","Data":"d6f7df5a19b2857f13af9223e8adde7de78f994949a01f1a98537fa3b5bff38f"}
Nov 01 20:30:13.541588 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.541533    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92gcr" event={"ID":"6559ca8c-d167-4f33-95f9-1856858fc3d5","Type":"ContainerStarted","Data":"4e87fb263650f7b3bdb607263dc55160c0598dd438963629d15bd4c9ce14a103"}
Nov 01 20:30:13.638392 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.638337    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f7pr8" podStartSLOduration=6.7316012480000005 podStartE2EDuration="11.638321546s" totalImagesPullingTime="4.906720298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:10.856841595 +0000 UTC m=+9.035229824" watchObservedRunningTime="2026-11-01 20:30:13.638321546 +0000 UTC m=+11.816709747"
Nov 01 20:30:13.869284 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:13.869187    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t2v7x" podStartSLOduration=3.693712991 podStartE2EDuration="11.869163912s" totalImagesPullingTime="8.175450921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:13.755679574 +0000 UTC m=+11.934067775" watchObservedRunningTime="2026-11-01 20:30:13.869163912 +0000 UTC m=+12.047552113"
Nov 01 20:30:14.545052 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.545021    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="e40b7a01122f01b8b2afb57eadc09cf9a6b830702b6412a77a290f1d10caf7c2" exitCode=0
Nov 01 20:30:14.545677 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.545116    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerDied","Data":"e40b7a01122f01b8b2afb57eadc09cf9a6b830702b6412a77a290f1d10caf7c2"}
Nov 01 20:30:14.549131 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.549071    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l7c7b" event={"ID":"5318ea66-6db7-4701-a20d-84b25e4c196c","Type":"ContainerStarted","Data":"526f3704535487d3ee01bf965898c4665cda029a95dc5b86bbfef49b6f9f7721"}
Nov 01 20:30:14.552828 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.552804    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq" event={"ID":"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd","Type":"ContainerStarted","Data":"3e800e98629db36c5abc9e3a845b98f4bff0fccc6faca5edd9e4474c6086417e"}
Nov 01 20:30:14.754786 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.754693    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-92gcr" podStartSLOduration=4.607554936 podStartE2EDuration="12.754675307s" totalImagesPullingTime="8.147120371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:13.868735227 +0000 UTC m=+12.047123429" watchObservedRunningTime="2026-11-01 20:30:14.754675307 +0000 UTC m=+12.933063515"
Nov 01 20:30:14.959846 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.959792    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l7c7b" podStartSLOduration=4.713354912 podStartE2EDuration="12.959776745s" totalImagesPullingTime="8.246421833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:14.844312496 +0000 UTC m=+13.022700697" watchObservedRunningTime="2026-11-01 20:30:14.959776745 +0000 UTC m=+13.138164945"
Nov 01 20:30:14.960044 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:14.960015    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq" podStartSLOduration=3.521010298 podStartE2EDuration="12.960008228s" totalImagesPullingTime="9.43899793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:14.959534513 +0000 UTC m=+13.137922714" watchObservedRunningTime="2026-11-01 20:30:14.960008228 +0000 UTC m=+13.138396429"
Nov 01 20:30:15.439010 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:15.438979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:15.439152 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:15.439072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:15.439152 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:15.439090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:15.439240 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:15.439188    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:15.439294 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:15.439280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:15.439377 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:15.439359    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:15.555356 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:15.555322    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xbb76" event={"ID":"428f74c3-acde-4cbb-afd8-32c13b30cadf","Type":"ContainerStarted","Data":"f0bbe8a4c15a7bc115e800f23164ad538e63ee9fbbde4290ce4618966a9609a9"}
Nov 01 20:30:15.671527 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:15.671155    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xbb76" podStartSLOduration=3.596556487 podStartE2EDuration="13.671137424s" totalImagesPullingTime="10.074580937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:15.670623098 +0000 UTC m=+13.849011299" watchObservedRunningTime="2026-11-01 20:30:15.671137424 +0000 UTC m=+13.849525625"
Nov 01 20:30:16.559810 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:16.559506    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="3ff2978bb97dc9bea40993aabb403f65e252a9421c05b2bf89bc54c5637193cd" exitCode=0
Nov 01 20:30:16.559810 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:16.559547    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerDied","Data":"3ff2978bb97dc9bea40993aabb403f65e252a9421c05b2bf89bc54c5637193cd"}
Nov 01 20:30:17.439593 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:17.439155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:17.439593 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:17.439179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:17.439593 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:17.439271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:17.439593 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:17.439275    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:17.439593 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:17.439367    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:17.439593 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:17.439435    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:18.566484 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:18.566449    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="0f375da57225f491fb201d1b7d01a2de801f25340f3bf56df9ce8743bff23fa1" exitCode=0
Nov 01 20:30:18.567000 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:18.566504    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerDied","Data":"0f375da57225f491fb201d1b7d01a2de801f25340f3bf56df9ce8743bff23fa1"}
Nov 01 20:30:18.578424 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:18.578401    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:18.578517 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:18.578507    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:30:18.578898 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:18.578880    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:19.232874 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:19.232845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:19.233050 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:19.232897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:19.233050 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.232975    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:19.233050 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.232986    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:19.233050 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.233020    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.233008702 +0000 UTC m=+33.411396881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:30:19.233050 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.233040    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret podName:1cdfd004-c495-4f69-9aba-35276b089b06 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.233025179 +0000 UTC m=+33.411413371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret") pod "global-pull-secret-syncer-8x2dv" (UID: "1cdfd004-c495-4f69-9aba-35276b089b06") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:30:19.333688 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:19.333663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:19.333788 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.333771    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:30:19.333788 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.333782    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:30:19.333857 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.333791    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxxmw for pod openshift-network-diagnostics/network-check-target-j4t6k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:19.333857 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.333831    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw podName:260814c3-535b-448a-9213-2b992657efb8 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.333818748 +0000 UTC m=+33.512206942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxxmw" (UniqueName: "kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw") pod "network-check-target-j4t6k" (UID: "260814c3-535b-448a-9213-2b992657efb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:30:19.438323 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:19.438303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:19.438418 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:19.438310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:19.438418 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.438388    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:19.438482 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:19.438310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:19.438482 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.438468    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:19.438541 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:19.438521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:20.573454 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.573279    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:20.574087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.573833    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" containerID="576bef084d5d240d58cf1b55d2e0276e0b70ef07ccb7b987f27675a0e263f4e9" exitCode=1
Nov 01 20:30:20.574087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.573885    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"7bca502abbe62ea3a65a0fe5c3bfbcbbcde99e8475d0f5d5f8cc64a781ad9d2e"}
Nov 01 20:30:20.574087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.573953    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"f25b18b4cc941ca8443588885f23ebf5d9968fd1310665a0f7fe746174f09e33"}
Nov 01 20:30:20.574087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.573993    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"8b03a9ad2f249c190325206feade605fc8eeafdb52db7c66bf272839c4512871"}
Nov 01 20:30:20.574087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.574029    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"ff725244a86447aa40892b961ee161e5855d295e2d45aeb8f444e71e1a99ffa5"}
Nov 01 20:30:20.574087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.574062    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerDied","Data":"576bef084d5d240d58cf1b55d2e0276e0b70ef07ccb7b987f27675a0e263f4e9"}
Nov 01 20:30:20.574285 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:20.574107    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"068e86def7edf6b8bb04ea86f5410f6ceb4bce0d3c93ad92a1855f747ae29dda"}
Nov 01 20:30:21.438707 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:21.438677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:21.438869 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:21.438777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:21.438869 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:21.438687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:21.438869 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:21.438678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:21.438869 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:21.438865    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:21.439083 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:21.438921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:21.578891 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:21.578863    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="45f81759989cfcb16d839265c09f03490955911e1fd6e895001de63e9d6bfe5d" exitCode=0
Nov 01 20:30:21.579264 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:21.578923    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerDied","Data":"45f81759989cfcb16d839265c09f03490955911e1fd6e895001de63e9d6bfe5d"}
Nov 01 20:30:22.583018 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:22.582980    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="b97cddfaa7af2f00f54ba7c3e5f599ccbe8bcdca8875f7837079ec0c9ea3513d" exitCode=0
Nov 01 20:30:22.583516 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:22.583055    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerDied","Data":"b97cddfaa7af2f00f54ba7c3e5f599ccbe8bcdca8875f7837079ec0c9ea3513d"}
Nov 01 20:30:22.586371 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:22.586352    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:22.586791 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:22.586770    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"51523ade6cdf7448a640dce84d1d15584834e2bd9b712e3ff98079ea3890697a"}
Nov 01 20:30:23.438976 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:23.438951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:23.438976 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:23.438961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:23.439132 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:23.439033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:23.439132 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:23.439095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:23.439218 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:23.439174    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:23.439248 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:23.439224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:23.592602 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:23.592571    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspt2" event={"ID":"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5","Type":"ContainerStarted","Data":"577e4fc197bbc3fa3f6546ff11cff5e951696fd1cf2699e191e0a47ff994fe35"}
Nov 01 20:30:23.709511 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:23.709437    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fspt2" podStartSLOduration=9.737423692 podStartE2EDuration="21.709424594s" totalImagesPullingTime="11.972000902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:23.709108709 +0000 UTC m=+21.887496909" watchObservedRunningTime="2026-11-01 20:30:23.709424594 +0000 UTC m=+21.887812794"
Nov 01 20:30:24.598485 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.598322    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:24.599044 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.598895    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"f9aa5d80fae4fab844afbba5b702952a8e48477c360c83e65df4d156a1978c33"}
Nov 01 20:30:24.599235 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.599213    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:24.599317 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.599267    2577 scope.go:122] "RemoveContainer" containerID="576bef084d5d240d58cf1b55d2e0276e0b70ef07ccb7b987f27675a0e263f4e9"
Nov 01 20:30:24.599538 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.599517    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:24.599688 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.599600    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:24.616058 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.616031    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:24.616145 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:24.616133    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:25.438883 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:25.438856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:25.439139 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:25.438862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:25.439139 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:25.438996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:25.439139 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:25.438995    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:25.439139 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:25.439058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:25.439301 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:25.439168    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:25.607199 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:25.607176    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:25.607686 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:25.607616    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" event={"ID":"9e3961b5-f2f0-4c47-9017-72de63a2f7c4","Type":"ContainerStarted","Data":"59618c8e3f79cd091c30ec01cc31b5ddad18eb551a4ca1144c3eee14d5408afe"}
Nov 01 20:30:25.730421 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:25.730343    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n" podStartSLOduration=8.902858498 podStartE2EDuration="23.730332576s" totalImagesPullingTime="14.827474078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:25.729947035 +0000 UTC m=+23.908335235" watchObservedRunningTime="2026-11-01 20:30:25.730332576 +0000 UTC m=+23.908720816"
Nov 01 20:30:26.752095 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:26.752048    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8x2dv"]
Nov 01 20:30:26.752598 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:26.752285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:26.752598 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:26.752415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:26.755796 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:26.755762    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cx7ds"]
Nov 01 20:30:26.755930 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:26.755884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:26.756040 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:26.755990    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:26.756429 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:26.756408    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4t6k"]
Nov 01 20:30:26.756506 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:26.756495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:26.756588 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:26.756569    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:28.441697 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:28.441623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:28.442022 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:28.441623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:28.442022 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:28.441730    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:28.442022 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:28.441623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:28.442022 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:28.441785    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:28.442022 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:28.441856    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:30.438880 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:30.438849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:30.439331 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:30.438961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:30.439331 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:30.438971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8x2dv" podUID="1cdfd004-c495-4f69-9aba-35276b089b06"
Nov 01 20:30:30.439331 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:30.439061    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx7ds" podUID="f99c6ff8-8987-46e6-bfb3-8aea67369ea1"
Nov 01 20:30:30.439331 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:30.439097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:30.439331 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:30.439149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4t6k" podUID="260814c3-535b-448a-9213-2b992657efb8"
Nov 01 20:30:31.665476 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.665319    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeReady"
Nov 01 20:30:31.665807 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.665544    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 20:30:31.777425 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.777400    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-75vgh"]
Nov 01 20:30:31.780719 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.780701    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pjl9s"]
Nov 01 20:30:31.780820 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.780807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:31.783970 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.783953    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-76pfm"]
Nov 01 20:30:31.784080 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.784065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:31.787352 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.787330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:31.822069 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.822046    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54cb57cf74-86lck"]
Nov 01 20:30:31.825242 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.825224    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9rgh"]
Nov 01 20:30:31.825336 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.825321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.828261 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.828240    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"]
Nov 01 20:30:31.828495 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.828475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.830770 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.830751    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"]
Nov 01 20:30:31.830872 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.830849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:31.833794 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.833778    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66ff46cf4c-rt6qk"]
Nov 01 20:30:31.833888 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.833876    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:31.837044 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.837024    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b48f6559f-rxmwf"]
Nov 01 20:30:31.837162 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.837145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:31.840613 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.840571    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"]
Nov 01 20:30:31.840726 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.840710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.844786 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.844769    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"]
Nov 01 20:30:31.844925 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.844908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:31.847972 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.847016    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"]
Nov 01 20:30:31.847972 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.847291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:31.852145 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.852128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:31.872772 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.872755    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"]
Nov 01 20:30:31.876908 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.876890    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"]
Nov 01 20:30:31.877013 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.876997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:31.878493 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.878472    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.878578 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.878508    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.878880 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.878865    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 20:30:31.878968 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.878900    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 20:30:31.879344 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.879331    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 20:30:31.880373 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.880357    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"]
Nov 01 20:30:31.880528 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.880481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:31.883792 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.883775    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-glm2w"]
Nov 01 20:30:31.883886 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.883872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:31.887687 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.887665    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dwl2n"]
Nov 01 20:30:31.887775 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.887722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:31.890796 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890782    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-75vgh"]
Nov 01 20:30:31.890863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890806    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-76pfm"]
Nov 01 20:30:31.890863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890819    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pjl9s"]
Nov 01 20:30:31.890863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890832    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54cb57cf74-86lck"]
Nov 01 20:30:31.890863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890844    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"]
Nov 01 20:30:31.890863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890857    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b48f6559f-rxmwf"]
Nov 01 20:30:31.891034 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890868    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"]
Nov 01 20:30:31.891034 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:31.891034 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.890881    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"]
Nov 01 20:30:31.891034 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.891020    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"]
Nov 01 20:30:31.891171 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.891051    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwl2n"]
Nov 01 20:30:31.923229 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.923209    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 20:30:31.923320 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.923273    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9chrm\""
Nov 01 20:30:31.923411 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.923389    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9m2k9\""
Nov 01 20:30:31.923487 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.923453    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 20:30:31.923554 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.923538    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-njcwg\""
Nov 01 20:30:31.923728 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.923542    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xls9n\""
Nov 01 20:30:31.924270 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.924251    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9rgh"]
Nov 01 20:30:31.924850 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.924831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.924950 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.924865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.924950 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.924892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.924950 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.924938    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"]
Nov 01 20:30:31.925679 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:31.925719 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b245f314-7357-4797-90d0-aa95f77b154a-serving-cert\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:31.925752 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.925784 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqc8\" (UniqueName: \"kubernetes.io/projected/83552290-2377-4a32-be4c-e1f10c121b12-kube-api-access-8bqc8\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:31.925814 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.925844 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-trusted-ca\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:31.925874 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfgg\" (UniqueName: \"kubernetes.io/projected/b245f314-7357-4797-90d0-aa95f77b154a-kube-api-access-tqfgg\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:31.925904 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336-kube-api-access-mf6wh\") pod \"kas-connection-checker-685895c7c9-76pfm\" (UID: \"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336\") " pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:31.925904 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cc3e550-0cb5-4301-82bc-17503dc53deb-serving-cert\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:31.925958 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.925988 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhdg\" (UniqueName: \"kubernetes.io/projected/5912e397-27c0-48e4-a25d-a902436d452d-kube-api-access-fxhdg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:31.926025 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.925992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.926062 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5912e397-27c0-48e4-a25d-a902436d452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:31.926062 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-default-certificate\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:31.926119 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-config\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:31.926119 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.926170 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.926170 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:31.926230 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:31.926230 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsrg\" (UniqueName: \"kubernetes.io/projected/bd018011-2b75-4090-812a-5b5039164aa9-kube-api-access-zzsrg\") pod \"kas-connection-checker-685895c7c9-75vgh\" (UID: \"bd018011-2b75-4090-812a-5b5039164aa9\") " pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:31.926294 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.926294 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrkv\" (UniqueName: \"kubernetes.io/projected/9cc3e550-0cb5-4301-82bc-17503dc53deb-kube-api-access-lwrkv\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:31.926294 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-snapshots\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.926384 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5912e397-27c0-48e4-a25d-a902436d452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:31.926384 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.926384 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/83552290-2377-4a32-be4c-e1f10c121b12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:31.926482 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhlw\" (UniqueName: \"kubernetes.io/projected/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-kube-api-access-fdhlw\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.926482 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.926482 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.926482 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.926667 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-serving-cert\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.926667 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9cc3e550-0cb5-4301-82bc-17503dc53deb-config\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:31.926766 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:31.926851 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.926904 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-stats-auth\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:31.926963 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.926950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2m9k\" (UniqueName: \"kubernetes.io/projected/b20d35a5-f97b-48a6-9dfb-73c6d656032c-kube-api-access-j2m9k\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:31.927231 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.927029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlm2n\" (UniqueName: \"kubernetes.io/projected/ace44fda-7418-4639-83db-e7499a8e140d-kube-api-access-wlm2n\") pod \"kas-connection-checker-685895c7c9-pjl9s\" (UID: \"ace44fda-7418-4639-83db-e7499a8e140d\") " pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:31.927396 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.927376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-tmp\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:31.927467 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.927448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrsg\" (UniqueName: \"kubernetes.io/projected/0b1356ae-ca5a-4464-8629-0272902c73cb-kube-api-access-mvrsg\") pod \"volume-data-source-validator-58c9f5f46d-rs7t5\" (UID: \"0b1356ae-ca5a-4464-8629-0272902c73cb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:31.927521 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.927504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.927582 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.927567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:31.932160 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.932137    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66ff46cf4c-rt6qk"]
Nov 01 20:30:31.932240 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.932219    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"]
Nov 01 20:30:31.933269 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.933249    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"]
Nov 01 20:30:31.934173 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934155    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"]
Nov 01 20:30:31.934479 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934460    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 20:30:31.934540 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934506    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.934540 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934519    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.934612 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934570    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.934670 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934637    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 20:30:31.934775 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934762    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.934835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934817    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.934835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934828    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.934925 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934894    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 20:30:31.934925 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934908    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 20:30:31.934925 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.934919    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 20:30:31.935068 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935032    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glm2w"]
Nov 01 20:30:31.935068 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935046    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 20:30:31.935171 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935155    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.935216 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935194    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 20:30:31.935266 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935231    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.935266 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935241    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.935355 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935264    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.935355 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935275    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.935355 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935300    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.935475 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935375    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 20:30:31.935530 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.935516    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:31.985785 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:31.985769    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 20:30:32.024542 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024519    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 20:30:32.024542 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024536    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 20:30:32.024685 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024582    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:32.024685 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024609    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 20:30:32.024786 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024687    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 20:30:32.024786 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024767    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 20:30:32.024889 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024814    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 20:30:32.024889 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024815    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 20:30:32.024889 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024859    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jgj4\""
Nov 01 20:30:32.025025 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.024985    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 20:30:32.025025 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025018    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 20:30:32.025104 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025018    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 20:30:32.025104 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025036    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 20:30:32.025104 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025017    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xct89\""
Nov 01 20:30:32.025104 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025067    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 20:30:32.025104 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025090    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 20:30:32.025104 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025095    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zmb7f\""
Nov 01 20:30:32.025274 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025149    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gnklv\""
Nov 01 20:30:32.025274 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025216    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n2j5c\""
Nov 01 20:30:32.025274 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025256    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pklm5\""
Nov 01 20:30:32.025364 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025319    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cpf8s\""
Nov 01 20:30:32.025396 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025369    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2nmd8\""
Nov 01 20:30:32.025460 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.025448    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-pmwqj\""
Nov 01 20:30:32.028536 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsrg\" (UniqueName: \"kubernetes.io/projected/bd018011-2b75-4090-812a-5b5039164aa9-kube-api-access-zzsrg\") pod \"kas-connection-checker-685895c7c9-75vgh\" (UID: \"bd018011-2b75-4090-812a-5b5039164aa9\") " pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:32.028585 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.028585 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrkv\" (UniqueName: \"kubernetes.io/projected/9cc3e550-0cb5-4301-82bc-17503dc53deb-kube-api-access-lwrkv\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.028585 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-snapshots\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.028694 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5912e397-27c0-48e4-a25d-a902436d452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.028694 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.028785 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/83552290-2377-4a32-be4c-e1f10c121b12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:32.028785 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0ba0752-5a6d-4490-a0bc-e3819c68a504-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:32.028884 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhlw\" (UniqueName: \"kubernetes.io/projected/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-kube-api-access-fdhlw\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.028884 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.028884 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.029021 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.029021 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-serving-cert\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.029021 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9cc3e550-0cb5-4301-82bc-17503dc53deb-config\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.029021 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028981    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.029021 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.028996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-stats-auth\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2m9k\" (UniqueName: \"kubernetes.io/projected/b20d35a5-f97b-48a6-9dfb-73c6d656032c-kube-api-access-j2m9k\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f5b52b-e610-4b2f-9521-aec1e69f2170-tmp-dir\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlm2n\" (UniqueName: \"kubernetes.io/projected/ace44fda-7418-4639-83db-e7499a8e140d-kube-api-access-wlm2n\") pod \"kas-connection-checker-685895c7c9-pjl9s\" (UID: \"ace44fda-7418-4639-83db-e7499a8e140d\") " pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-tmp\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.029257 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrsg\" (UniqueName: \"kubernetes.io/projected/0b1356ae-ca5a-4464-8629-0272902c73cb-kube-api-access-mvrsg\") pod \"volume-data-source-validator-58c9f5f46d-rs7t5\" (UID: \"0b1356ae-ca5a-4464-8629-0272902c73cb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029357    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-snapshots\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5912e397-27c0-48e4-a25d-a902436d452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgd6z\" (UniqueName: \"kubernetes.io/projected/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-kube-api-access-rgd6z\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b245f314-7357-4797-90d0-aa95f77b154a-serving-cert\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.029565    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.029582    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.029630 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqc8\" (UniqueName: \"kubernetes.io/projected/83552290-2377-4a32-be4c-e1f10c121b12-kube-api-access-8bqc8\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.029678    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.529639726 +0000 UTC m=+30.708027919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhfqf\" (UniqueName: \"kubernetes.io/projected/25f5b52b-e610-4b2f-9521-aec1e69f2170-kube-api-access-xhfqf\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-trusted-ca\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029563    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/83552290-2377-4a32-be4c-e1f10c121b12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfgg\" (UniqueName: \"kubernetes.io/projected/b245f314-7357-4797-90d0-aa95f77b154a-kube-api-access-tqfgg\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336-kube-api-access-mf6wh\") pod \"kas-connection-checker-685895c7c9-76pfm\" (UID: \"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336\") " pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cc3e550-0cb5-4301-82bc-17503dc53deb-serving-cert\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029869    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-tmp\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhdg\" (UniqueName: \"kubernetes.io/projected/5912e397-27c0-48e4-a25d-a902436d452d-kube-api-access-fxhdg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.029964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5912e397-27c0-48e4-a25d-a902436d452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.030362 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-default-certificate\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ks6\" (UniqueName: \"kubernetes.io/projected/b60f90ff-66ee-410f-b953-aee5ed9b8d47-kube-api-access-c7ks6\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfrt\" (UniqueName: \"kubernetes.io/projected/dd13ffc1-85a5-476f-83f8-3dac47e667a8-kube-api-access-xsfrt\") pod \"network-check-source-6f78d59f99-jm5qf\" (UID: \"dd13ffc1-85a5-476f-83f8-3dac47e667a8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-config\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5b52b-e610-4b2f-9521-aec1e69f2170-config-volume\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60f90ff-66ee-410f-b953-aee5ed9b8d47-cert\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030323    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9cc3e550-0cb5-4301-82bc-17503dc53deb-config\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.030599    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.030613    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.030684    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.530650757 +0000 UTC m=+30.709038964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:32.031103 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.030811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.030983    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.530968881 +0000 UTC m=+30.709357079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.031286    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.031358    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.031365    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.031486    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.031520    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.531502346 +0000 UTC m=+30.709890534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:32.031682 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.031617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.032130 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.032076    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:32.032251 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.032228    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-trusted-ca\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.032314 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.032276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-config\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.032368 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.032354    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.532339251 +0000 UTC m=+30.710727433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:32.034247 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.034194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b245f314-7357-4797-90d0-aa95f77b154a-serving-cert\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.034772 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.034261    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cc3e550-0cb5-4301-82bc-17503dc53deb-serving-cert\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.034772 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.034291    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-stats-auth\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.034772 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.034550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.034772 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.034618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-serving-cert\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.034772 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.034726    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.035367 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035206    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 20:30:32.035367 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035208    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 20:30:32.035367 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035289    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 20:30:32.035367 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035321    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:32.035367 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035323    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 20:30:32.035367 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035353    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 20:30:32.035648 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.035648 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035506    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.035779 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.035738    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5912e397-27c0-48e4-a25d-a902436d452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.036024 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.036007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-default-certificate\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.086434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.086415    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsrg\" (UniqueName: \"kubernetes.io/projected/bd018011-2b75-4090-812a-5b5039164aa9-kube-api-access-zzsrg\") pod \"kas-connection-checker-685895c7c9-75vgh\" (UID: \"bd018011-2b75-4090-812a-5b5039164aa9\") " pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:32.086566 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.086466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrkv\" (UniqueName: \"kubernetes.io/projected/9cc3e550-0cb5-4301-82bc-17503dc53deb-kube-api-access-lwrkv\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.089091 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.089060    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlm2n\" (UniqueName: \"kubernetes.io/projected/ace44fda-7418-4639-83db-e7499a8e140d-kube-api-access-wlm2n\") pod \"kas-connection-checker-685895c7c9-pjl9s\" (UID: \"ace44fda-7418-4639-83db-e7499a8e140d\") " pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:32.090835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.090835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336-kube-api-access-mf6wh\") pod \"kas-connection-checker-685895c7c9-76pfm\" (UID: \"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336\") " pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:32.090835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090206    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2m9k\" (UniqueName: \"kubernetes.io/projected/b20d35a5-f97b-48a6-9dfb-73c6d656032c-kube-api-access-j2m9k\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.090835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.090835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090503    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrsg\" (UniqueName: \"kubernetes.io/projected/0b1356ae-ca5a-4464-8629-0272902c73cb-kube-api-access-mvrsg\") pod \"volume-data-source-validator-58c9f5f46d-rs7t5\" (UID: \"0b1356ae-ca5a-4464-8629-0272902c73cb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:32.090835 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090738    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.091091 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.090952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.091366 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.091340    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqc8\" (UniqueName: \"kubernetes.io/projected/83552290-2377-4a32-be4c-e1f10c121b12-kube-api-access-8bqc8\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:32.091626 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.091609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhlw\" (UniqueName: \"kubernetes.io/projected/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-kube-api-access-fdhlw\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.092164 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.092142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhdg\" (UniqueName: \"kubernetes.io/projected/5912e397-27c0-48e4-a25d-a902436d452d-kube-api-access-fxhdg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.092267 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.092257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfgg\" (UniqueName: \"kubernetes.io/projected/b245f314-7357-4797-90d0-aa95f77b154a-kube-api-access-tqfgg\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.092944 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.092925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:32.099221 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.099202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:32.104102 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.104073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:32.130902 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.130859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.130902 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.130899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhfqf\" (UniqueName: \"kubernetes.io/projected/25f5b52b-e610-4b2f-9521-aec1e69f2170-kube-api-access-xhfqf\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.131134 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.130969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ks6\" (UniqueName: \"kubernetes.io/projected/b60f90ff-66ee-410f-b953-aee5ed9b8d47-kube-api-access-c7ks6\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.131134 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.130991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfrt\" (UniqueName: \"kubernetes.io/projected/dd13ffc1-85a5-476f-83f8-3dac47e667a8-kube-api-access-xsfrt\") pod \"network-check-source-6f78d59f99-jm5qf\" (UID: \"dd13ffc1-85a5-476f-83f8-3dac47e667a8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:32.131134 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.131045    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:32.131134 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.131125    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.63110193 +0000 UTC m=+30.809490112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:32.131345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:32.131345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5b52b-e610-4b2f-9521-aec1e69f2170-config-volume\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.131345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60f90ff-66ee-410f-b953-aee5ed9b8d47-cert\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.131345 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.131315    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:32.131345 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0ba0752-5a6d-4490-a0bc-e3819c68a504-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:32.131614 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.131361    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.631347808 +0000 UTC m=+30.809735990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:32.131614 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.131614 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f5b52b-e610-4b2f-9521-aec1e69f2170-tmp-dir\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.131614 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.131614 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd6z\" (UniqueName: \"kubernetes.io/projected/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-kube-api-access-rgd6z\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.131966 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.131948    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5b52b-e610-4b2f-9521-aec1e69f2170-config-volume\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.132229 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.132055    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0ba0752-5a6d-4490-a0bc-e3819c68a504-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:32.132229 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.132106    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:32.132229 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.132190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f5b52b-e610-4b2f-9521-aec1e69f2170-tmp-dir\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.132229 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.132211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:32.632195123 +0000 UTC m=+30.810583316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:32.132574 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.132550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.135381 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.135337    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60f90ff-66ee-410f-b953-aee5ed9b8d47-cert\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.144921 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.144740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:32.161984 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.161264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:32.183585 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.182810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:32.192222 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.187356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:32.192222 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.191711    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ks6\" (UniqueName: \"kubernetes.io/projected/b60f90ff-66ee-410f-b953-aee5ed9b8d47-kube-api-access-c7ks6\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.195776 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.193957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:32.195776 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.195335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhfqf\" (UniqueName: \"kubernetes.io/projected/25f5b52b-e610-4b2f-9521-aec1e69f2170-kube-api-access-xhfqf\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.199639 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.199591    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfrt\" (UniqueName: \"kubernetes.io/projected/dd13ffc1-85a5-476f-83f8-3dac47e667a8-kube-api-access-xsfrt\") pod \"network-check-source-6f78d59f99-jm5qf\" (UID: \"dd13ffc1-85a5-476f-83f8-3dac47e667a8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:32.202720 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.202687    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd6z\" (UniqueName: \"kubernetes.io/projected/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-kube-api-access-rgd6z\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.212432 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.212246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:32.232782 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.232356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:32.438415 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.438388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:32.438516 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.438388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:32.438694 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.438669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:32.481096 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.481075    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-75vgh"]
Nov 01 20:30:32.485052 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.485033    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-76pfm"]
Nov 01 20:30:32.487407 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.487384    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b44fcb4_4d02_4b52_b0bb_b67dcaafd336.slice/crio-f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501 WatchSource:0}: Error finding container f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501: Status 404 returned error can't find the container with id f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501
Nov 01 20:30:32.491017 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.491001    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 20:30:32.491389 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.491377    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pskvw\""
Nov 01 20:30:32.491434 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.491412    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 20:30:32.491468 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.491437    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cg54x\""
Nov 01 20:30:32.491516 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.491503    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kzbx4\""
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.535419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.535464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.535567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.535610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.535681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.535837    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.535853    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.535901    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.535884167 +0000 UTC m=+31.714272360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.535970    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536006    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.535995611 +0000 UTC m=+31.714383804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536059    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536087    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.536077398 +0000 UTC m=+31.714465592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536141    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536150    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536179    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.536168128 +0000 UTC m=+31.714556325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:32.536228 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.536201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.536184301 +0000 UTC m=+31.714572496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:32.546493 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.546419    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9rgh"]
Nov 01 20:30:32.581018 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.580952    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"]
Nov 01 20:30:32.599649 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.599620    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb245f314_7357_4797_90d0_aa95f77b154a.slice/crio-60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77 WatchSource:0}: Error finding container 60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77: Status 404 returned error can't find the container with id 60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77
Nov 01 20:30:32.611862 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.611844    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pjl9s"]
Nov 01 20:30:32.613779 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.613728    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace44fda_7418_4639_83db_e7499a8e140d.slice/crio-caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688 WatchSource:0}: Error finding container caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688: Status 404 returned error can't find the container with id caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688
Nov 01 20:30:32.626161 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.626133    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9rgh" event={"ID":"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b","Type":"ContainerStarted","Data":"22a1e9321c0ae4b5f17f55e12e88b2e9f51c37476bdaa5459e07273dacc85327"}
Nov 01 20:30:32.627803 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.627778    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-76pfm" event={"ID":"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336","Type":"ContainerStarted","Data":"67a7cc5f9e2871221a5585e3b94eb21b84e2cbfe79ce6006df8e2890d63a4340"}
Nov 01 20:30:32.627869 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.627825    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-76pfm" event={"ID":"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336","Type":"ContainerStarted","Data":"f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501"}
Nov 01 20:30:32.629555 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.629440    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pjl9s" event={"ID":"ace44fda-7418-4639-83db-e7499a8e140d","Type":"ContainerStarted","Data":"caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688"}
Nov 01 20:30:32.631028 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.631008    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"]
Nov 01 20:30:32.634732 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.634705    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-75vgh" event={"ID":"bd018011-2b75-4090-812a-5b5039164aa9","Type":"ContainerStarted","Data":"7b9257995fa91fe3dff7feb4631cade4dd045353ac899922fe11485210d42134"}
Nov 01 20:30:32.634808 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.634748    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-75vgh" event={"ID":"bd018011-2b75-4090-812a-5b5039164aa9","Type":"ContainerStarted","Data":"0abb6e40ac8ad5c0038e0f3925831f51c702e1171f8048d8d685dd40ca4c1723"}
Nov 01 20:30:32.636101 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.636073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:32.636179 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.636126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:32.636240 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.636204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:32.636288 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.636248    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:32.636333 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.636303    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.636283961 +0000 UTC m=+31.814672154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:32.636333 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.636313    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:32.636431 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.636364    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:32.636431 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.636422    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.636374009 +0000 UTC m=+31.814762188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:32.636535 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:32.636485    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:33.636431539 +0000 UTC m=+31.814819720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:32.636588 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.636545    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1356ae_ca5a_4464_8629_0272902c73cb.slice/crio-7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65 WatchSource:0}: Error finding container 7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65: Status 404 returned error can't find the container with id 7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65
Nov 01 20:30:32.636781 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.636756    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" event={"ID":"b245f314-7357-4797-90d0-aa95f77b154a","Type":"ContainerStarted","Data":"60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77"}
Nov 01 20:30:32.638711 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.638645    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"]
Nov 01 20:30:32.639519 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.639498    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"]
Nov 01 20:30:32.643725 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.643703    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd13ffc1_85a5_476f_83f8_3dac47e667a8.slice/crio-f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284 WatchSource:0}: Error finding container f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284: Status 404 returned error can't find the container with id f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284
Nov 01 20:30:32.644336 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.644316    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc3e550_0cb5_4301_82bc_17503dc53deb.slice/crio-6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61 WatchSource:0}: Error finding container 6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61: Status 404 returned error can't find the container with id 6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61
Nov 01 20:30:32.664231 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.664211    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"]
Nov 01 20:30:32.665957 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.665936    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5912e397_27c0_48e4_a25d_a902436d452d.slice/crio-a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8 WatchSource:0}: Error finding container a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8: Status 404 returned error can't find the container with id a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8
Nov 01 20:30:32.666493 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:32.666464    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwl2n"]
Nov 01 20:30:32.669294 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:32.669273    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60f90ff_66ee_410f_b953_aee5ed9b8d47.slice/crio-93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a WatchSource:0}: Error finding container 93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a: Status 404 returned error can't find the container with id 93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a
Nov 01 20:30:33.545416 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.545384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:33.545559 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.545423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:33.545559 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545511    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:33.545559 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545512    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:33.545559 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.545541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:33.545559 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545559    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.545547287 +0000 UTC m=+33.723935483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545581    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.545575367 +0000 UTC m=+33.723963546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545607    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545617    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.545642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545651    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.54564067 +0000 UTC m=+33.724028853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.545693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545699    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545707    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:33.545739 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545731    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.54572244 +0000 UTC m=+33.724110648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:33.546012 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.545784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.545777097 +0000 UTC m=+33.724165277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:33.641251 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.641219    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5" event={"ID":"0b1356ae-ca5a-4464-8629-0272902c73cb","Type":"ContainerStarted","Data":"7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65"}
Nov 01 20:30:33.642650 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.642597    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf" event={"ID":"dd13ffc1-85a5-476f-83f8-3dac47e667a8","Type":"ContainerStarted","Data":"f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284"}
Nov 01 20:30:33.644240 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.644184    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k" event={"ID":"9cc3e550-0cb5-4301-82bc-17503dc53deb","Type":"ContainerStarted","Data":"6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61"}
Nov 01 20:30:33.645868 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.645840    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pjl9s" event={"ID":"ace44fda-7418-4639-83db-e7499a8e140d","Type":"ContainerStarted","Data":"eee39c16876c801db04103b3932dae5a834f68cf8823c9f15756804fa9ce81c9"}
Nov 01 20:30:33.646099 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.646079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:33.646176 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.646163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:33.646241 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.646198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:33.646291 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.646250    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:33.646334 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.646297    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:33.646334 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.646320    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.646302163 +0000 UTC m=+33.824690359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:33.646414 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.646342    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.646327995 +0000 UTC m=+33.824716188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:33.646414 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.646296    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:33.646414 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:33.646377    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:35.646367824 +0000 UTC m=+33.824756029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:33.647181 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.647159    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn" event={"ID":"5912e397-27c0-48e4-a25d-a902436d452d","Type":"ContainerStarted","Data":"a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8"}
Nov 01 20:30:33.648544 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.648523    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwl2n" event={"ID":"b60f90ff-66ee-410f-b953-aee5ed9b8d47","Type":"ContainerStarted","Data":"93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a"}
Nov 01 20:30:33.971571 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:33.969848    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-75vgh" podStartSLOduration=442.969831148 podStartE2EDuration="7m22.969831148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:33.969272831 +0000 UTC m=+32.147661029" watchObservedRunningTime="2026-11-01 20:30:33.969831148 +0000 UTC m=+32.148219351"
Nov 01 20:30:34.202505 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:34.201247    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pjl9s" podStartSLOduration=443.201234041 podStartE2EDuration="7m23.201234041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:34.083874968 +0000 UTC m=+32.262263173" watchObservedRunningTime="2026-11-01 20:30:34.201234041 +0000 UTC m=+32.379622608"
Nov 01 20:30:34.202505 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:34.201691    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-76pfm" podStartSLOduration=443.201682512 podStartE2EDuration="7m23.201682512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:34.198062947 +0000 UTC m=+32.376451147" watchObservedRunningTime="2026-11-01 20:30:34.201682512 +0000 UTC m=+32.380070714"
Nov 01 20:30:35.263006 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.262968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:35.263346 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.263109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:35.263569 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.263543    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 20:30:35.263693 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.263621    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:31:07.263602444 +0000 UTC m=+65.441990643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : secret "metrics-daemon-secret" not found
Nov 01 20:30:35.276571 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.276518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:35.364588 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.364251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:35.367415 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.367332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:35.456641 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.456509    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:35.460642 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.460618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:35.565530 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.565465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:35.565788 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.565560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:35.565788 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.565590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:35.565788 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565669    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:35.565788 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565693    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:35.565788 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.565706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:35.565788 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565759    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.565737648 +0000 UTC m=+37.744125843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.565803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565826    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.565863135 +0000 UTC m=+37.744251330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565902    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565938    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565949    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.565940    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.565927646 +0000 UTC m=+37.744315844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.566039    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.566026511 +0000 UTC m=+37.744414696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:35.566095 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.566054    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.56604536 +0000 UTC m=+37.744433546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:35.664700 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.664644    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5" event={"ID":"0b1356ae-ca5a-4464-8629-0272902c73cb","Type":"ContainerStarted","Data":"dbc5eab4bda57d111fa58ba4a7c8b9ccd27e89475b969f2d9110e7ad34e4fff7"}
Nov 01 20:30:35.667132 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.667105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:35.667263 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.667205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:35.667324 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.667261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:35.667423 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.667400    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:35.667497 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.667451    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.667436301 +0000 UTC m=+37.845824485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:35.667560 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.667507    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:35.667560 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.667540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.667529508 +0000 UTC m=+37.845917693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:35.667684 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.667591    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:35.667684 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:35.667634    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.667610274 +0000 UTC m=+37.845998459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:35.891378 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.891278    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5" podStartSLOduration=415.247091585 podStartE2EDuration="6m57.891259136s" totalImagesPullingTime="2.644167551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:35.779780183 +0000 UTC m=+33.958168383" watchObservedRunningTime="2026-11-01 20:30:35.891259136 +0000 UTC m=+34.069647339"
Nov 01 20:30:35.892533 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.892508    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4t6k"]
Nov 01 20:30:35.940877 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:35.939852    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8x2dv"]
Nov 01 20:30:35.943091 ip-10-0-131-188 kubenswrapper[2577]: W1101 20:30:35.943063    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdfd004_c495_4f69_9aba_35276b089b06.slice/crio-0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14 WatchSource:0}: Error finding container 0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14: Status 404 returned error can't find the container with id 0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14
Nov 01 20:30:36.669915 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:36.669837    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8x2dv" event={"ID":"1cdfd004-c495-4f69-9aba-35276b089b06","Type":"ContainerStarted","Data":"0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14"}
Nov 01 20:30:36.672020 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:36.671951    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4t6k" event={"ID":"260814c3-535b-448a-9213-2b992657efb8","Type":"ContainerStarted","Data":"70e8e8de594f6a049743d385a21a5cf00a72309b24b25ec0c93a420e94ee6128"}
Nov 01 20:30:37.683754 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:37.682956    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9rgh" event={"ID":"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b","Type":"ContainerStarted","Data":"a95c4325610e143250e71b12ce5457f36e709afdf05238d7477dacd7481630e2"}
Nov 01 20:30:37.804692 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:37.804630    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-v9rgh" podStartSLOduration=415.441465376 podStartE2EDuration="6m59.80461561s" totalImagesPullingTime="4.363150234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:37.802690698 +0000 UTC m=+35.981078901" watchObservedRunningTime="2026-11-01 20:30:37.80461561 +0000 UTC m=+35.983003812"
Nov 01 20:30:38.424309 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.423016    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ntncn"]
Nov 01 20:30:38.444576 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.444547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.485516 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.485452    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntncn"]
Nov 01 20:30:38.532150 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.532125    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 20:30:38.532379 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.532364    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8z8c\""
Nov 01 20:30:38.532453 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.532409    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 20:30:38.596390 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.596367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.596489 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.596424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10cbc89e-6b29-48e4-8218-03be6f3e3992-crio-socket\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.596551 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.596528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9xg\" (UniqueName: \"kubernetes.io/projected/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-api-access-7j9xg\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.596608 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.596557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.596608 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.596601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10cbc89e-6b29-48e4-8218-03be6f3e3992-data-volume\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.690689 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.690614    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" event={"ID":"b245f314-7357-4797-90d0-aa95f77b154a","Type":"ContainerStarted","Data":"ec93ab611c4dbd85d56e16b1ebaef980ec21edb89a54b3a69390abfab4960401"}
Nov 01 20:30:38.692019 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.691961    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:38.693627 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.693412    2577 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-9m5fw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Nov 01 20:30:38.693627 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.693449    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.697601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10cbc89e-6b29-48e4-8218-03be6f3e3992-crio-socket\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.697719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9xg\" (UniqueName: \"kubernetes.io/projected/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-api-access-7j9xg\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.697798    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10cbc89e-6b29-48e4-8218-03be6f3e3992-crio-socket\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.697790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.697851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10cbc89e-6b29-48e4-8218-03be6f3e3992-data-volume\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:38.697901    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.697915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:38.697947    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.197930895 +0000 UTC m=+37.376319099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:38.697962    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:38.697994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:39.197984064 +0000 UTC m=+37.376372270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : configmap "kube-rbac-proxy" not found
Nov 01 20:30:38.698327 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.698238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10cbc89e-6b29-48e4-8218-03be6f3e3992-data-volume\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.703468 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.703408    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k" event={"ID":"9cc3e550-0cb5-4301-82bc-17503dc53deb","Type":"ContainerStarted","Data":"513369be3d6e94f3e79e7bcae3e3481a1db33a8ad3a2219ea661eab71b9e09a5"}
Nov 01 20:30:38.755967 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.755940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9xg\" (UniqueName: \"kubernetes.io/projected/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-api-access-7j9xg\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:38.817546 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.813275    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podStartSLOduration=414.839695111 podStartE2EDuration="7m0.813252178s" totalImagesPullingTime="5.973557067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:38.806945576 +0000 UTC m=+36.985333778" watchObservedRunningTime="2026-11-01 20:30:38.813252178 +0000 UTC m=+36.991640383"
Nov 01 20:30:38.924735 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:38.924647    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k" podStartSLOduration=415.672293947 podStartE2EDuration="7m0.924632396s" totalImagesPullingTime="5.252338449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:38.924355937 +0000 UTC m=+37.102744139" watchObservedRunningTime="2026-11-01 20:30:38.924632396 +0000 UTC m=+37.103020596"
Nov 01 20:30:39.202051 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.202021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:39.202185 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.202164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:39.202257 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.202230    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:30:39.202308 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.202294    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.202273097 +0000 UTC m=+38.380661297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : configmap "kube-rbac-proxy" not found
Nov 01 20:30:39.202376 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.202361    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:39.202423 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.202418    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.202403929 +0000 UTC m=+38.380792122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.605235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.605335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.605364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.605453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.605486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605578    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605622    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.605610135 +0000 UTC m=+45.783998314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605771    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.605758589 +0000 UTC m=+45.784146783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605841    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:39.605863 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605853    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:39.606491 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605882    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.60587197 +0000 UTC m=+45.784260168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:39.606491 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605932    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:39.606491 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605940    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:39.606491 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.605966    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.60595705 +0000 UTC m=+45.784345243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:39.606491 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.606016    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:39.606491 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.606047    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.606037156 +0000 UTC m=+45.784425348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:39.706716 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.706680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.706777    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.706809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.706838    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.706820481 +0000 UTC m=+45.885208675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.706886    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.706905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.706923    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.706912366 +0000 UTC m=+45.885300553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.706962    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:39.707149 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:39.706998    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.706989223 +0000 UTC m=+45.885377418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:39.708684 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.708260    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf" event={"ID":"dd13ffc1-85a5-476f-83f8-3dac47e667a8","Type":"ContainerStarted","Data":"1d8fa09f737ab90c773ccdfbfa960ffbb1a5bbc0becb4c3b26a46170540bc0c2"}
Nov 01 20:30:39.710404 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.710387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/0.log"
Nov 01 20:30:39.710521 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.710421    2577 generic.go:356] "Generic (PLEG): container finished" podID="b245f314-7357-4797-90d0-aa95f77b154a" containerID="ec93ab611c4dbd85d56e16b1ebaef980ec21edb89a54b3a69390abfab4960401" exitCode=255
Nov 01 20:30:39.710521 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.710503    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" event={"ID":"b245f314-7357-4797-90d0-aa95f77b154a","Type":"ContainerDied","Data":"ec93ab611c4dbd85d56e16b1ebaef980ec21edb89a54b3a69390abfab4960401"}
Nov 01 20:30:39.710686 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.710673    2577 scope.go:122] "RemoveContainer" containerID="ec93ab611c4dbd85d56e16b1ebaef980ec21edb89a54b3a69390abfab4960401"
Nov 01 20:30:39.712624 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.712596    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn" event={"ID":"5912e397-27c0-48e4-a25d-a902436d452d","Type":"ContainerStarted","Data":"5e253f536b9757f87e238576941e5981d01f1d2d98efaa61f0617856f6c2b251"}
Nov 01 20:30:39.714173 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.714112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwl2n" event={"ID":"b60f90ff-66ee-410f-b953-aee5ed9b8d47","Type":"ContainerStarted","Data":"d4031c63dba879e56de87daad485e40ae4acb29ac14c103665164a5d38d4a5d6"}
Nov 01 20:30:39.824857 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:39.824802    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf" podStartSLOduration=414.35878304 podStartE2EDuration="7m0.824788758s" totalImagesPullingTime="6.466005718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:39.824089635 +0000 UTC m=+38.002477835" watchObservedRunningTime="2026-11-01 20:30:39.824788758 +0000 UTC m=+38.003176958"
Nov 01 20:30:40.058548 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.058499    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn" podStartSLOduration=415.628216359 podStartE2EDuration="7m2.058481759s" totalImagesPullingTime="6.4302654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:40.057189808 +0000 UTC m=+38.235578009" watchObservedRunningTime="2026-11-01 20:30:40.058481759 +0000 UTC m=+38.236869960"
Nov 01 20:30:40.212930 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.212896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:40.213258 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.213015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:40.213258 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:40.213018    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:30:40.213258 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:40.213088    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.213067985 +0000 UTC m=+40.391456183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : configmap "kube-rbac-proxy" not found
Nov 01 20:30:40.213258 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:40.213132    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:40.213258 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:40.213191    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.213175605 +0000 UTC m=+40.391563799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:40.720223 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.720158    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4t6k" event={"ID":"260814c3-535b-448a-9213-2b992657efb8","Type":"ContainerStarted","Data":"e4feccc93b5b4529f910a3cafaee518c7559c426cd3854870baf3674603fd4c0"}
Nov 01 20:30:40.720625 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.720326    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:40.722174 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.722149    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/1.log"
Nov 01 20:30:40.722675 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.722633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/0.log"
Nov 01 20:30:40.722783 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.722690    2577 generic.go:356] "Generic (PLEG): container finished" podID="b245f314-7357-4797-90d0-aa95f77b154a" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0" exitCode=255
Nov 01 20:30:40.722783 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.722775    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" event={"ID":"b245f314-7357-4797-90d0-aa95f77b154a","Type":"ContainerDied","Data":"a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0"}
Nov 01 20:30:40.722898 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.722816    2577 scope.go:122] "RemoveContainer" containerID="ec93ab611c4dbd85d56e16b1ebaef980ec21edb89a54b3a69390abfab4960401"
Nov 01 20:30:40.723024 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.723007    2577 scope.go:122] "RemoveContainer" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0"
Nov 01 20:30:40.723214 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:40.723191    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9m5fw_openshift-console-operator(b245f314-7357-4797-90d0-aa95f77b154a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a"
Nov 01 20:30:40.838557 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.838511    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dwl2n" podStartSLOduration=3.933538491 podStartE2EDuration="9.83849788s" totalImagesPullingTime="5.904959389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:40.174364396 +0000 UTC m=+38.352752598" watchObservedRunningTime="2026-11-01 20:30:40.83849788 +0000 UTC m=+39.016886082"
Nov 01 20:30:40.956352 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:40.956296    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j4t6k" podStartSLOduration=34.799394509 podStartE2EDuration="38.956277177s" totalImagesPullingTime="4.156882668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:40.838044724 +0000 UTC m=+39.016432924" watchObservedRunningTime="2026-11-01 20:30:40.956277177 +0000 UTC m=+39.134665378"
Nov 01 20:30:41.728177 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:41.728151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/1.log"
Nov 01 20:30:41.728876 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:41.728851    2577 scope.go:122] "RemoveContainer" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0"
Nov 01 20:30:41.729229 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:41.729187    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9m5fw_openshift-console-operator(b245f314-7357-4797-90d0-aa95f77b154a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a"
Nov 01 20:30:42.165614 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.165575    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:42.234939 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.234908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:42.235087 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.234991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:42.235087 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:42.235070    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:42.235194 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:42.235095    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:30:42.235194 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:42.235145    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:46.235125216 +0000 UTC m=+44.413513406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:42.235194 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:42.235167    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:46.235155717 +0000 UTC m=+44.413543898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : configmap "kube-rbac-proxy" not found
Nov 01 20:30:42.733108 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.733087    2577 scope.go:122] "RemoveContainer" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0"
Nov 01 20:30:42.733443 ip-10-0-131-188 kubenswrapper[2577]: E1101 20:30:42.733260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9m5fw_openshift-console-operator(b245f314-7357-4797-90d0-aa95f77b154a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a"
Nov 01 20:30:42.733729 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.733709    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8x2dv" event={"ID":"1cdfd004-c495-4f69-9aba-35276b089b06","Type":"ContainerStarted","Data":"75a4d503660bcc1c725a7db309c0d8b87d3144443ef2a40cb4c5aa8fecc5c289"}
Nov 01 20:30:42.847504 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.847418    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8x2dv" podStartSLOduration=34.19693167 podStartE2EDuration="40.847405378s" totalImagesPullingTime="6.650473708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:42.846334089 +0000 UTC m=+41.024722291" watchObservedRunningTime="2026-11-01 20:30:42.847405378 +0000 UTC m=+41.025793601"
Nov 01 20:30:42.891118 ip-10-0-131-188 kubenswrapper[2577]: I1101 20:30:42.891094    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:30:42.891257 ip-10-0-131-188 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 20:30:42.911001 ip-10-0-131-188 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 20:30:42.911160 ip-10-0-131-188 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 20:30:42.911210 ip-10-0-131-188 systemd[1]: kubelet.service: Consumed 2.727s CPU time.
Nov 01 20:30:42.922206 ip-10-0-131-188 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 20:30:42.996765 ip-10-0-131-188 kubenswrapper[4872]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:30:42.996765 ip-10-0-131-188 kubenswrapper[4872]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 20:30:42.996765 ip-10-0-131-188 kubenswrapper[4872]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:30:42.996765 ip-10-0-131-188 kubenswrapper[4872]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:30:42.998702 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998621    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:42.998702 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998702    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998707    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998712    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998716    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998721    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998724    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998729    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998733    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998736    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998738    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998741    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998744    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998747    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998749    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998751    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998754    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998757    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998759    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998762    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:42.998771 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998764    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998767    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998770    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998772    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998781    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998784    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998787    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998790    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998792    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998795    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998797    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998800    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998802    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998805    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998807    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998810    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998812    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998815    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998817    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:42.999226 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998820    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998822    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998824    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998827    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998829    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998831    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998834    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998836    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998838    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998841    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998843    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998846    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998848    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998850    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998853    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998856    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998859    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998862    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998865    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998867    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:42.999709 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998870    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998872    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998875    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998877    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998880    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998882    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998885    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998887    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998890    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998892    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998895    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998897    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998900    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998903    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998905    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998908    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998910    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998912    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998915    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998917    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.000189 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998920    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998922    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998925    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998927    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998930    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998932    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998935    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998939    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998941    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998945    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998949    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998952    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998955    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998957    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998960    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998962    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998964    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998967    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998969    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998971    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.000667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998974    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998976    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998979    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998981    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998984    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998986    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.998990    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999713    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999719    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999722    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999725    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999728    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999730    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999733    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999736    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999739    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999742    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999744    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999747    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.001165 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999750    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999752    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999755    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999758    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999760    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999768    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999771    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999774    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999776    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999779    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999781    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999784    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999786    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999789    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999791    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999794    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999797    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999799    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999803    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.001667 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999805    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999808    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999811    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999813    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999816    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999819    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999821    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999824    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999826    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999828    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999831    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999833    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999836    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999839    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999841    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999844    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999846    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999849    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999851    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999859    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.002139 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999861    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999864    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999866    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999869    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999871    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999874    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999876    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999879    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999882    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999884    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999886    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999889    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999892    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999894    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999897    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999899    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999902    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999904    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999907    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.002627 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999909    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999911    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999914    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999916    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999919    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999922    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999924    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999927    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999929    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999931    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999934    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999936    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999938    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999941    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999943    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999946    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999948    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999950    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999953    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999955    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.003088 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999958    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999960    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999963    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999965    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999968    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999970    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999974    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999979    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999984    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999988    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999990    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999993    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999996    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:42.999999    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000002    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000005    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000482    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000489    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000492    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.003574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000495    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000498    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000500    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000503    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000505    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000508    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000511    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000514    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000516    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000518    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000521    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000524    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000527    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000529    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000532    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000534    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000538    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000542    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000546    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000548    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.004041 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000551    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000554    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000556    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000559    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000561    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000564    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000566    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000569    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000571    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000574    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000577    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000580    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000582    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000585    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000587    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000590    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000592    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000595    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000597    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.004553 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000601    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000603    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000606    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000609    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000611    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000614    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000616    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000618    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000621    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000623    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000626    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000628    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000630    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000633    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000635    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000638    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000640    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000643    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000645    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.005015 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000648    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000664    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000669    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000672    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000678    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000682    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000686    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000689    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000693    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000696    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000699    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000702    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000704    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000708    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000714    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000717    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000720    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000725    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000729    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.005468 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000733    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000736    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000738    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000741    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000743    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000746    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000748    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000751    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000753    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000756    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000758    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000761    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000763    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000766    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000768    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000771    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000774    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000777    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000779    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000782    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.005959 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000784    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000787    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000789    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000792    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000794    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.000797    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000889    4872 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000895    4872 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000904    4872 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000910    4872 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000916    4872 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000921    4872 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000927    4872 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000933    4872 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000952    4872 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000959    4872 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000962    4872 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000966    4872 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000969    4872 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000972    4872 flags.go:64] FLAG: --cgroup-root=""
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000975    4872 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000978    4872 flags.go:64] FLAG: --client-ca-file=""
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000981    4872 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 20:30:43.006432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000984    4872 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000989    4872 flags.go:64] FLAG: --cluster-domain=""
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000992    4872 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000996    4872 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.000999    4872 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001003    4872 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001007    4872 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001010    4872 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001013    4872 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001016    4872 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001019    4872 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001022    4872 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001025    4872 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001029    4872 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001032    4872 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001036    4872 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001039    4872 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001042    4872 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001046    4872 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001049    4872 flags.go:64] FLAG: --enable-server="true"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001052    4872 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001057    4872 flags.go:64] FLAG: --event-burst="100"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001060    4872 flags.go:64] FLAG: --event-qps="50"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001063    4872 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 20:30:43.007014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001066    4872 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001068    4872 flags.go:64] FLAG: --eviction-hard=""
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001073    4872 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001076    4872 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001079    4872 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001082    4872 flags.go:64] FLAG: --eviction-soft=""
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001085    4872 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001087    4872 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001090    4872 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001093    4872 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001096    4872 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001099    4872 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001102    4872 flags.go:64] FLAG: --feature-gates=""
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001106    4872 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001109    4872 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001112    4872 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001115    4872 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001118    4872 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001121    4872 flags.go:64] FLAG: --help="false"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001124    4872 flags.go:64] FLAG: --hostname-override="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001127    4872 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001130    4872 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001133    4872 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 20:30:43.007751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001137    4872 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001140    4872 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001143    4872 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001147    4872 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001150    4872 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001153    4872 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001156    4872 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001159    4872 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001169    4872 flags.go:64] FLAG: --kube-reserved=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001173    4872 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001176    4872 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001179    4872 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001182    4872 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001184    4872 flags.go:64] FLAG: --lock-file=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001187    4872 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001190    4872 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001193    4872 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001198    4872 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001201    4872 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001204    4872 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001207    4872 flags.go:64] FLAG: --logging-format="text"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001210    4872 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001213    4872 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001216    4872 flags.go:64] FLAG: --manifest-url=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001219    4872 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 20:30:43.008687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001223    4872 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001226    4872 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001230    4872 flags.go:64] FLAG: --max-pods="110"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001233    4872 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001236    4872 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001239    4872 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001242    4872 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001245    4872 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001248    4872 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001251    4872 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001258    4872 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001262    4872 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001265    4872 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001268    4872 flags.go:64] FLAG: --pod-cidr=""
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001271    4872 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001274    4872 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001277    4872 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001280    4872 flags.go:64] FLAG: --port="10250"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001282    4872 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001285    4872 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a2325141049c29c0"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001288    4872 flags.go:64] FLAG: --qos-reserved=""
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001291    4872 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001294    4872 flags.go:64] FLAG: --register-node="true"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001296    4872 flags.go:64] FLAG: --register-with-taints=""
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001300    4872 flags.go:64] FLAG: --registry-burst="10"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001303    4872 flags.go:64] FLAG: --registry-qps="5"
Nov 01 20:30:43.009769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001305    4872 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001308    4872 flags.go:64] FLAG: --reserved-memory=""
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001312    4872 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001315    4872 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001318    4872 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001320    4872 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001323    4872 flags.go:64] FLAG: --runonce="false"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001326    4872 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001329    4872 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001332    4872 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001336    4872 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001338    4872 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001342    4872 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001345    4872 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001348    4872 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001350    4872 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001354    4872 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001357    4872 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001360    4872 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001363    4872 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001366    4872 flags.go:64] FLAG: --system-cgroups=""
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001369    4872 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001374    4872 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001377    4872 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001380    4872 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 20:30:43.010936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001385    4872 flags.go:64] FLAG: --tls-min-version=""
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001387    4872 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001390    4872 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001393    4872 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001396    4872 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001399    4872 flags.go:64] FLAG: --v="2"
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001403    4872 flags.go:64] FLAG: --version="false"
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001407    4872 flags.go:64] FLAG: --vmodule=""
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001411    4872 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001414    4872 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001544    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001549    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001552    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001555    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001557    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001560    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001563    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001565    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001568    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001570    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001573    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001575    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.012247 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001578    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001583    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001585    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001588    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001590    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001593    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001596    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001598    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001601    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001603    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001606    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001608    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001611    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001613    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001616    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001618    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001620    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001623    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001626    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.013029 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001628    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001630    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001633    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001635    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001638    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001640    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001643    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001645    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001647    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001650    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001668    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001672    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001675    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001677    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001682    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001684    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001687    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001689    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001692    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001695    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.013558 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001698    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001700    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001703    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001705    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001708    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001710    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001712    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001714    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001717    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001719    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001723    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001726    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001728    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001731    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001733    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001736    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001738    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001741    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001743    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.014183 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001746    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001748    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001751    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001753    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001756    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001760    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001763    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001768    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001771    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001774    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001778    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001781    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001785    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001787    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001790    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001793    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001795    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001798    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001801    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.015170 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001803    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001805    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001808    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001810    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001814    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001817    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001819    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001822    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001825    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001827    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001830    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001832    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001835    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001838    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001840    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001843    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.015829 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001845    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.001871    4872 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001905    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001910    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001915    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001919    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001922    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001925    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001927    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001930    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001932    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001936    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001938    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001941    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001943    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.016311 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001946    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001948    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001951    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001953    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001956    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001958    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001962    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001965    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001967    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001969    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001972    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001975    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001977    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001981    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001983    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001986    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001988    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001991    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001993    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001996    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.016744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.001998    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002002    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002005    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002007    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002010    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002012    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002014    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002017    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002020    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002023    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002026    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002028    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002031    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002033    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002036    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002038    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002040    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002043    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002046    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.017220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002049    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002051    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002054    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002056    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002059    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002061    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002064    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002066    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002069    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002071    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002073    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002076    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002078    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002081    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002085    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002087    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002089    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002092    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002094    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002097    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.017676 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002099    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002102    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002104    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002107    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002109    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002112    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002115    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002117    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002119    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002122    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002124    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002128    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002131    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002133    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002136    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002138    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002141    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002143    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002146    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002149    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.018141 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002151    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002154    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002156    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002159    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002161    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002164    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002167    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002170    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002172    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002175    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002179    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002183    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002186    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.002188    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.005907    4872 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.005924    4872 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.005940    4872 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.005945    4872 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.005999    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.018604 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006004    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006008    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006013    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006016    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006019    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006022    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006024    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006027    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006030    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006033    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006035    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006038    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006040    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006043    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006046    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006049    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006051    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006055    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006058    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.019061 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006060    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006063    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006066    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006068    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006071    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006073    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006076    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006084    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006087    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006095    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006098    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006101    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006103    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006106    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006108    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006111    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006113    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006115    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006118    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006120    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.019505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006123    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006125    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006128    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006130    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006133    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006135    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006138    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006141    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006143    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006146    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006148    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006151    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006154    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006156    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006159    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006161    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006164    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006166    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006170    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.019977 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006173    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006176    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006178    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006187    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006190    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006193    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006196    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006198    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006201    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006203    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006205    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006208    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006210    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006212    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006215    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006217    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006220    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006222    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006225    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.020429 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006227    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006229    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006232    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006234    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006237    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006239    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006241    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006244    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006246    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006248    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006251    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006253    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006256    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006258    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006261    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006263    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006265    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006273    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006276    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.020918 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006278    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006281    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006283    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006286    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006288    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006291    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006293    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006296    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006298    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.006323    4872 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006491    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006496    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006499    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006502    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006505    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006507    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006510    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.021367 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006513    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006515    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006519    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006521    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006524    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006526    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006529    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006531    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006534    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006538    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006541    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006543    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006546    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006548    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006559    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006563    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006567    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006570    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006572    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.021768 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006575    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006577    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006580    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006582    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006585    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006587    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006590    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006592    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006595    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006597    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006600    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006602    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006605    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006607    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006610    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006612    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006615    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006617    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006620    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006622    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.022233 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006625    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006627    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006630    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006632    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006635    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006638    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006642    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006644    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006664    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006669    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006672    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006675    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006677    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006680    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006682    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006685    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006688    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006690    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006693    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006695    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.022716 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006698    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006700    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006703    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006705    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006708    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006710    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006713    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006716    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006718    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006721    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006723    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006726    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006728    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006731    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006733    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006736    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006738    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006741    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006744    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006746    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.023173 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006749    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006758    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006761    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006764    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006766    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006769    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006771    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006774    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006777    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006781    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006785    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006789    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006793    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006796    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006799    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006801    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006805    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006808    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006810    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.023631 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006813    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.006837    4872 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006874    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006879    4872 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006882    4872 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006885    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006887    4872 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006890    4872 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006892    4872 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006895    4872 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006898    4872 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006900    4872 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006903    4872 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006905    4872 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006908    4872 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006917    4872 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:30:43.024151 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006921    4872 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006926    4872 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006929    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006932    4872 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006935    4872 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006938    4872 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006940    4872 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006943    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006946    4872 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006948    4872 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006951    4872 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006953    4872 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006956    4872 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006960    4872 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006962    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006965    4872 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006967    4872 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006970    4872 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006972    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006975    4872 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:30:43.024571 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006977    4872 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006980    4872 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006982    4872 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006990    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006993    4872 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006995    4872 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.006998    4872 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007000    4872 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007003    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007005    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007008    4872 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007011    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007019    4872 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007022    4872 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007025    4872 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007028    4872 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007030    4872 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007033    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007035    4872 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007038    4872 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:30:43.025127 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007040    4872 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007043    4872 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007045    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007048    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007051    4872 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007054    4872 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007056    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007059    4872 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007061    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007064    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007066    4872 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007069    4872 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007071    4872 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007074    4872 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007076    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007079    4872 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007081    4872 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007083    4872 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007086    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:30:43.025603 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007088    4872 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007091    4872 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007093    4872 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007096    4872 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007099    4872 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007101    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007109    4872 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007112    4872 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007114    4872 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007117    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007119    4872 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007123    4872 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007126    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007129    4872 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007132    4872 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007134    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007136    4872 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007139    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007142    4872 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:30:43.026099 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007144    4872 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007147    4872 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007149    4872 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007152    4872 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007154    4872 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007157    4872 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007159    4872 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007161    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007163    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007166    4872 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007168    4872 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007171    4872 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007173    4872 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:43.007175    4872 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.007232    4872 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.009622    4872 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.009710    4872 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.009963    4872 server.go:1014] "Starting client certificate rotation"
Nov 01 20:30:43.026666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.010104    4872 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.010158    4872 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 20:25:02 +0000 UTC" deadline="2028-03-29 15:24:42.243487141 +0000 UTC"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.010194    4872 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12330h53m59.233296249s"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.010691    4872 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.012335    4872 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.013477    4872 log.go:25] "Validated CRI v1 runtime API"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.015442    4872 log.go:25] "Validated CRI v1 image API"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.016169    4872 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 20:30:43.027359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.017878    4872 fs.go:135] Filesystem UUIDs: map[4cbcd29a-6d9f-429c-b003-a24b39241eee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d91d2445-defb-4f2c-8605-5b36e046690e:/dev/nvme0n1p3]
Nov 01 20:30:43.027900 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.017897    4872 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0abb6e40ac8ad5c0038e0f3925831f51c702e1171f8048d8d685dd40ca4c1723/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0abb6e40ac8ad5c0038e0f3925831f51c702e1171f8048d8d685dd40ca4c1723/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22a1e9321c0ae4b5f17f55e12e88b2e9f51c37476bdaa5459e07273dacc85327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22a1e9321c0ae4b5f17f55e12e88b2e9f51c37476bdaa5459e07273dacc85327/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70e8e8de594f6a049743d385a21a5cf00a72309b24b25ec0c93a420e94ee6128/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70e8e8de594f6a049743d385a21a5cf00a72309b24b25ec0c93a420e94ee6128/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c04f0fc524901f8a7537b7c1c8c1bed1cd6f595100a32d48494684ecec0bd7cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c04f0fc524901f8a7537b7c1c8c1bed1cd6f595100a32d48494684ecec0bd7cd/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b1356ae-ca5a-4464-8629-0272902c73cb/volumes/kubernetes.io~projected/kube-api-access-mvrsg:{mountpoint:/var/lib/kubelet/pods/0b1356ae-ca5a-4464-8629-0272902c73cb/volumes/kubernetes.io~projected/kube-api-access-mvrsg major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10cbc89e-6b29-48e4-8218-03be6f3e3992/volumes/kubernetes.io~projected/kube-api-access-7j9xg:{mountpoint:/var/lib/kubelet/pods/10cbc89e-6b29-48e4-8218-03be6f3e3992/volumes/kubernetes.io~projected/kube-api-access-7j9xg major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cdfd004-c495-4f69-9aba-35276b089b06/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1cdfd004-c495-4f69-9aba-35276b089b06/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25f5b52b-e610-4b2f-9521-aec1e69f2170/volumes/kubernetes.io~projected/kube-api-access-xhfqf:{mountpoint:/var/lib/kubelet/pods/25f5b52b-e610-4b2f-9521-aec1e69f2170/volumes/kubernetes.io~projected/kube-api-access-xhfqf major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260814c3-535b-448a-9213-2b992657efb8/volumes/kubernetes.io~projected/kube-api-access-lxxmw:{mountpoint:/var/lib/kubelet/pods/260814c3-535b-448a-9213-2b992657efb8/volumes/kubernetes.io~projected/kube-api-access-lxxmw major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26fed350-454f-4f57-8989-016d524f5b4f/volumes/kubernetes.io~projected/kube-api-access-lp849:{mountpoint:/var/lib/kubelet/pods/26fed350-454f-4f57-8989-016d524f5b4f/volumes/kubernetes.io~projected/kube-api-access-lp849 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~projected/kube-api-access-dqqmj:{mountpoint:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~projected/kube-api-access-dqqmj major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32264030-2487-4a8d-b5e4-3f233831760b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/32264030-2487-4a8d-b5e4-3f233831760b/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~projected/kube-api-access-wktzx:{mountpoint:/var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~projected/kube-api-access-wktzx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5318ea66-6db7-4701-a20d-84b25e4c196c/volumes/kubernetes.io~projected/kube-api-access-zv9nk:{mountpoint:/var/lib/kubelet/pods/5318ea66-6db7-4701-a20d-84b25e4c196c/volumes/kubernetes.io~projected/kube-api-access-zv9nk major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5912e397-27c0-48e4-a25d-a902436d452d/volumes/kubernetes.io~projected/kube-api-access-fxhdg:{mountpoint:/var/lib/kubelet/pods/5912e397-27c0-48e4-a25d-a902436d452d/volumes/kubernetes.io~projected/kube-api-access-fxhdg major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5912e397-27c0-48e4-a25d-a902436d452d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5912e397-27c0-48e4-a25d-a902436d452d/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6559ca8c-d167-4f33-95f9-1856858fc3d5/volumes/kubernetes.io~projected/kube-api-access-b5277:{mountpoint:/var/lib/kubelet/pods/6559ca8c-d167-4f33-95f9-1856858fc3d5/volumes/kubernetes.io~projected/kube-api-access-b5277 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5/volumes/kubernetes.io~projected/kube-api-access-frfzs:{mountpoint:/var/lib/kubelet/pods/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5/volumes/kubernetes.io~projected/kube-api-access-frfzs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd/volumes/kubernetes.io~projected/kube-api-access-q4lcr:{mountpoint:/var/lib/kubelet/pods/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd/volumes/kubernetes.io~projected/kube-api-access-q4lcr major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83552290-2377-4a32-be4c-e1f10c121b12/volumes/kubernetes.io~projected/kube-api-access-8bqc8:{mountpoint:/var/lib/kubelet/pods/83552290-2377-4a32-be4c-e1f10c121b12/volumes/kubernetes.io~projected/kube-api-access-8bqc8 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~projected/kube-api-access-5psvg:{mountpoint:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~projected/kube-api-access-5psvg major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b/volumes/kubernetes.io~projected/kube-api-access-fdhlw:{mountpoint:/var/lib/kubelet/pods/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b/volumes/kubernetes.io~projected/kube-api-access-fdhlw major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336/volumes/kubernetes.io~projected/kube-api-access-mf6wh:{mountpoint:/var/lib/kubelet/pods/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336/volumes/kubernetes.io~projected/kube-api-access-mf6wh major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc3e550-0cb5-4301-82bc-17503dc53deb/volumes/kubernetes.io~projected/kube-api-access-lwrkv:{mountpoint:/var/lib/kubelet/pods/9cc3e550-0cb5-4301-82bc-17503dc53deb/volumes/kubernetes.io~projected/kube-api-access-lwrkv major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc3e550-0cb5-4301-82bc-17503dc53deb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9cc3e550-0cb5-4301-82bc-17503dc53deb/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volumes/kubernetes.io~projected/kube-api-access-4x8wh:{mountpoint:/var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volumes/kubernetes.io~projected/kube-api-access-4x8wh major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ace44fda-7418-4639-83db-e7499a8e140d/volumes/kubernetes.io~projected/kube-api-access-wlm2n:{mountpoint:/var/lib/kubelet/pods/ace44fda-7418-4639-83db-e7499a8e140d/volumes/kubernetes.io~projected/kube-api-access-wlm2n major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~projected/kube-api-access-j2m9k:{mountpoint:/var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~projected/kube-api-access-j2m9k major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b245f314-7357-4797-90d0-aa95f77b154a/volumes/kubernetes.io~projected/kube-api-access-tqfgg:{mountpoint:/var/lib/kubelet/pods/b245f314-7357-4797-90d0-aa95f77b154a/volumes/kubernetes.io~projected/kube-api-access-tqfgg major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b245f314-7357-4797-90d0-aa95f77b154a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b245f314-7357-4797-90d0-aa95f77b154a/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b60f90ff-66ee-410f-b953-aee5ed9b8d47/volumes/kubernetes.io~projected/kube-api-access-c7ks6:{mountpoint:/var/lib/kubelet/pods/b60f90ff-66ee-410f-b953-aee5ed9b8d47/volumes/kubernetes.io~projected/kube-api-access-c7ks6 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b60f90ff-66ee-410f-b953-aee5ed9b8d47/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b60f90ff-66ee-410f-b953-aee5ed9b8d47/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd018011-2b75-4090-812a-5b5039164aa9/volumes/kubernetes.io~projected/kube-api-access-zzsrg:{mountpoint:/var/lib/kubelet/pods/bd018011-2b75-4090-812a-5b5039164aa9/volumes/kubernetes.io~projected/kube-api-access-zzsrg major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c136bbc2-2ebc-40b7-8548-ca250b2dba7c/volumes/kubernetes.io~projected/kube-api-access-rgd6z:{mountpoint:/var/lib/kubelet/pods/c136bbc2-2ebc-40b7-8548-ca250b2dba7c/volumes/kubernetes.io~projected/kube-api-access-rgd6z major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce97c7cb-9ccb-468e-af69-a8f8e922ec24/volumes/kubernetes.io~projected/kube-api-access-zgdqs:{mountpoint:/var/lib/kubelet/pods/ce97c7cb-9ccb-468e-af69-a8f8e922ec24/volumes/kubernetes.io~projected/kube-api-access-zgdqs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd13ffc1-85a5-476f-83f8-3dac47e667a8/volumes/kubernetes.io~projected/kube-api-access-xsfrt:{mountpoint:/var/lib/kubelet/pods/dd13ffc1-85a5-476f-83f8-3dac47e667a8/volumes/kubernetes.io~projected/kube-api-access-xsfrt major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f99c6ff8-8987-46e6-bfb3-8aea67369ea1/volumes/kubernetes.io~projected/kube-api-access-zkxpg:{mountpoint:/var/lib/kubelet/pods/f99c6ff8-8987-46e6-bfb3-8aea67369ea1/volumes/kubernetes.io~projected/kube-api-access-zkxpg major:0 minor:53 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/84cbb0ce55892533ffac35fe77f9672ee17fd95d5d6bb6b4c57b95e065728205/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/669ca0263f8f1a9dbbf1321dee690fecd3d7a98decdcf3272677b9a5fb9251c6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a563f5bc3ce7bc312da12a6e7328d45c3f8f223c9428b2ebe5f3ed49e379313f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/4b3069f8031a44e9a8943087e31f4ac2bb2c39fcf2d0fff6742b5361b0d6138e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/2190fcde4a49077a4b3c17f3ee8cd4cf1a0a9f77b3700cf6aff876b4dca3e73a/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d5eb1d37770b61ad098d6d984915eb0e0bea7985d953a146440e28c55035837a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9b2ce2018d07cdc7de4b8d0b3ad710f3762e0b732f19e22e43651c93ee11993a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9d18905a5fdd1fd2996775954bc66cbb8b3f05771e9334c2618936391e752725/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/856a3a3f8536d6a69f676f120111a27ea52cb95575ac698e1ade14426e45457a/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/a53612dda84b725d32ff5e1dc8b38dc4877055896178d64ac223a5ddc9f1af51/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/b3d22559afb3ae859641772d8e8e3d1ac1bf9b42e94a98703cb0a37294da30d2/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f84370811ee809653674ff11c67969412fe2f4ea5e1fa7bafcc30a039823eeeb/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5046636d50b2c80db4088501cfecf486dd6e3638cdd9ef4837ca3f9dde277f5a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/58456690ea0b38aa4d257457ef39b54e09b20874bcf2535426b8cf3d1d3a09fe/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f5c7ba64edcb70a2afb03703a25eab8705034a653c5ac8a54cca64d55f573a5e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3929346a747e8a51def46bbc7b55d79e51a1ccc216eceefc909d2e0a7397345f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f23d738a2b2bb69f6863c7516e5837c8e506c69872109be3a1fc286d667c0579/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ad7898f890683d4c58aea0bc40befde06590502392ade06c6d5a8e64a21726fe/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bfa26e5314ad0feff696963d50ce308993e0691e87c32456c86fc46ce344095a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/69ac9682facf2e552b104da1b7c791fdb878b2d00d1fd394294ade99cd1c2c0a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/495ebf2ee4366a88e3900b619d4ebb8bcfd2964cc57276bd08f2bf429e93e4c6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5125a1963130364528430654ab83bd40cf543d2d32ec8eb03c233c758a69634a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8e531c1e50b3c0e4b33e481e8415d27ae3d14111f941c20860246628b04bc821/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/7b524a3b110683082ab82dd7f0c2d3c8a51ceaeb3ee0fa2135aacb049911d406/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/707981ceae01e43e740fdd42c4334f6e40927bb4e38607ce25f216604d2ae25a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a4532f974fe968d903e570c4881a015eb5452ebddcd070159b401655b33e57d3/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/52a2b28e6a764e46ceb9d68a649589f36861700d027a6af201ed5f6efcc50061/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/574f8faad3135cc1716e6f6ce345811f5cf6cd511ed32588df46da9ff2152507/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/3d60a6959bfe2579331057ba28a48151a7b4b15f4ad78f68a54e2809b0785ec0/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/85b98c3aa9cfb3573812f8c945bd39ecad2c17eeb57428feff2f496b13f0c0eb/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/db066940cb14dfb1ea34822e5d92457fbea770622ceeef2471795756abe763b3/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c7c6ec76b21260f4ef5d73ba3dbd0df4656d7f4fba18c31ff0fd496a993fb24a/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/1b0612e6a345168c1b1a71df8730471475eed566764ebe3a507a1b42f559a94e/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0295dca1e3e920f989e69be902cf77b8aae2614941420e3212277cecdfb1c282/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/0e90a91aa6b21cb10f69a93f3c3a3f4d790c9cd3d34aff841cafe53930cdafb7/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/fd07e08dd9f5e94074418b25b42b2c0125e912d373973aa046bc364e51612384/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/c3cb499ee76226a6256671b9db449cc89600f43c02efde229edec64a2bd059d1/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/3e15060ee1e8c54cdc9bb0224a6d660f48bb9ed44efaff7a71f40de4a365d208/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/01577ee22d6e8dba6d88558bc20adbfac9be6d6505c97cd3bd836701b7c21d4b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/0d87152e6fc0d122459ae40093327169d2f6dd7d00553873afeef8ccdff17e04/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/c578e403c65dcde73d5c8b068b50c5fe91b4b5066f954d3ed2b1023b49448204/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/84f5c311c6da8e3b988d73469fb1eb60f5ac5f2162e350bd5e6e7d9d91ae17a2/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/2fd5f9bb8734654f7cabd22d649946f67ad5f84fd85f36573eebf147e13e42b1/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/d02387971b0ce1ef7deb33eb971888cd99f239e26f5cbe9f0985f939600636d9/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/fc1d45832eb4b8bdaf7aa1fe3400f479415772217d8c0ec412c87db7788a815d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e47892b6a785a248be4027f4f88aa365f1292159ad122c9169422448fdaa79e2/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5b8b8e37f521a149df8238b28a60c7a67d3b67b36a7cb1e0bdca8e62c258abd3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/912d4923d6b4e3c36a563fcc5f9341215c0f93623e560c7ea0b71ffda1b63a2f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5191758a8cbebdae59afdbcc8c0f653c7c14f56fa6156e1bb4b2028a680746dd/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6ff05bbdf1c0d6a35c6097ccbb8638ffd8b055d18d5bc99c812b618781ad4607/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/746793b852d4f30712200b86ca9fa42c24e8a127be554b761361c93bc11336d6/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c9ebc5749a5b1e3fb609c0a7c537140e7249d607d41fd48fe6f38fd9d2af2e45/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/49ea3cdeca4986aa3657e019d07b606c678a37f4f03dfd050462988e2d95bd97/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/d2486e09e1d8c86fbc778e0999318c9398c86190aeeda6b09723cfe3e3d3d182/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.025281    4872 manager.go:217] Machine: {Timestamp:2026-11-01 20:30:43.024826597 +0000 UTC m=+0.073936406 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198588 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24ce3f1506e987e9029f3a8eb3a53d SystemUUID:ec24ce3f-1506-e987-e902-9f3a8eb3a53d BootID:142d24da-a6ed-4812-bce3-794c5ad81ecf Filesystems:[{Device:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/260814c3-535b-448a-9213-2b992657efb8/volumes/kubernetes.io~projected/kube-api-access-lxxmw DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5912e397-27c0-48e4-a25d-a902436d452d/volumes/kubernetes.io~projected/kube-api-access-fxhdg DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6559ca8c-d167-4f33-95f9-1856858fc3d5/volumes/kubernetes.io~projected/kube-api-access-b5277 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc3e550-0cb5-4301-82bc-17503dc53deb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b1356ae-ca5a-4464-8629-0272902c73cb/volumes/kubernetes.io~projected/kube-api-access-mvrsg DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~projected/kube-api-access-dqqmj DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70e8e8de594f6a049743d385a21a5cf00a72309b24b25ec0c93a420e94ee6128/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c8c616394772ddfd547371ae1883cd63885980d6e33a5ec36421ee571970e86/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b/volumes/kubernetes.io~projected/kube-api-access-fdhlw DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd018011-2b75-4090-812a-5b5039164aa9/volumes/kubernetes.io~projected/kube-api-access-zzsrg DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3a8e6fedb115f616bdef83f41d324e37dd375a8f6a56db1a74725751d32e284/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5912e397-27c0-48e4-a25d-a902436d452d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ace44fda-7418-4639-83db-e7499a8e140d/volumes/kubernetes.io~projected/kube-api-access-wlm2n DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caf9248d57c69f6a4fe85bb13715d6a05f37340c3d39f79b245c2ad7806be688/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5c643cc83805295fe8aaf869716df36cf08004ff5aa956dbdd3a89870815501/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a080ffc306a20514b2e9ee658605f085a72a594795161f4910526dc25e2dc65/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5318ea66-6db7-4701-a20d-84b25e4c196c/volumes/kubernetes.io~projected/kube-api-access-zv9nk DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0abb6e40ac8ad5c0038e0f3925831f51c702e1171f8048d8d685dd40ca4c1723/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25f5b52b-e610-4b2f-9521-aec1e69f2170/volumes/kubernetes.io~projected/kube-api-access-xhfqf DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93ef8f71e4b861ab83b1e0fa2145bd166193a135bd181d8e8ebc32a3ec53793a/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26fed350-454f-4f57-8989-016d524f5b4f/volumes/kubernetes.io~projected/kube-api-access-lp849 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e005b2f58641d9032ca17d5abf8cfa587fc687b61acfcf646a957e15d095ee4d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22a1e9321c0ae4b5f17f55e12e88b2e9f51c37476bdaa5459e07273dacc85327/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60c16bc36fb8e209e29f6ac5c316500ee1e1979678e9331c8369ed90a7ff0c77/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b245f314-7357-4797-90d0-aa95f77b154a/volumes/kubernetes.io~projected/kube-api-access-tqfgg DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dd13ffc1-85a5-476f-83f8-3dac47e667a8/volumes/kubernetes.io~projected/kube-api-access-xsfrt DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eb07f89b513e838dc2d1394665e0a4d907b5723678e9bf28d3fb0ba064c7a61/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9cd9b9940657315290c221c3b2421213f01b28d981ec7f3e12c745dcd12f420/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83552290-2377-4a32-be4c-e1f10c121b12/volumes/kubernetes.io~projected/kube-api-access-8bqc8 DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ab0a120c29e9a89c8c3deab0b09968db8457ce5b2628bca1c81f5ae8d132609/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b245f314-7357-4797-90d0-aa95f77b154a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336/volumes/kubernetes.io~projected/kube-api-access-mf6wh DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cdfd004-c495-4f69-9aba-35276b089b06/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72d123740074b58c31d05e6d798d5ec8ac13ee239954b11503ab734be9e149d0/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96f1f7a037055511c32a24d668fcfd96f74689255cd7b57666255e3042ee1f88/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b60f90ff-66ee-410f-b953-aee5ed9b8d47/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f99c6ff8-8987-46e6-bfb3-8aea67369ea1/volumes/kubernetes.io~projected/kube-api-access-zkxpg DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce97c7cb-9ccb-468e-af69-a8f8e922ec24/volumes/kubernetes.io~projected/kube-api-access-zgdqs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84723caf514bc5e75cc64324ce7dc484b9f9627c2eff5621c7e3495a92e35919/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e3961b5-f2f0-4c47-9017-72de63a2f7c4/volumes/kubernetes.io~projected/kube-api-access-4x8wh DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b60f90ff-66ee-410f-b953-aee5ed9b8d47/volumes/kubernetes.io~projected/kube-api-access-c7ks6 DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/428f74c3-acde-4cbb-afd8-32c13b30cadf/volumes/kubernetes.io~projected/kube-api-access-wktzx DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc3e550-0cb5-4301-82bc-17503dc53deb/volumes/kubernetes.io~projected/kube-api-access-lwrkv DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d848a8ae59469256bdcb1e35c807a2c1e29306cff6b8af78fcf9a3b8739fd14/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c04f0fc524901f8a7537b7c1c8c1bed1cd6f595100a32d48494684ecec0bd7cd/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b20d35a5-f97b-48a6-9dfb-73c6d656032c/volumes/kubernetes.io~projected/kube-api-access-j2m9k DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10cbc89e-6b29-48e4-8218-03be6f3e3992/volumes/kubernetes.io~projected/kube-api-access-7j9xg DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes/kubernetes.io~projected/kube-api-access-5psvg DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c136bbc2-2ebc-40b7-8548-ca250b2dba7c/volumes/kubernetes.io~projected/kube-api-access-rgd6z DeviceMajor:0 DeviceMinor:237 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a171d48997fc9f833edab8510fbe7a49014b3f29e8415fe14edb646dedae07c8/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/32264030-2487-4a8d-b5e4-3f233831760b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd/volumes/kubernetes.io~projected/kube-api-access-q4lcr DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5/volumes/kubernetes.io~projected/kube-api-access-frfzs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0abb6e40ac8ad5c MacAddress:6e:bd:b8:c9:58:24 Speed:10000 Mtu:8901} {Name:0d848a8ae594692 MacAddress:5e:1f:2f:6c:94:45 Speed:10000 Mtu:8901} {Name:22a1e9321c0ae4b MacAddress:9a:b2:c8:3a:8e:62 Speed:10000 Mtu:8901} {Name:60c16bc36fb8e20 MacAddress:f2:f3:5b:8d:e1:d7 Speed:10000 Mtu:8901} {Name:6eb07f89b513e83 MacAddress:02:e0:6d:90:31:f4 Speed:10000 Mtu:8901} {Name:70e8e8de594f6a0 MacAddress:3e:de:6f:fb:ba:89 Speed:10000 Mtu:8901} {Name:7a080ffc306a205 MacAddress:d2:ea:4e:1d:09:cf Speed:10000 Mtu:8901} {Name:93ef8f71e4b861a MacAddress:ea:63:30:d4:0e:6d Speed:10000 Mtu:8901} {Name:a171d48997fc9f8 MacAddress:26:18:ef:93:bc:04 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:38:33:7c:81:03 Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:e9:bb:46:10:76 Speed:0 Mtu:8901} {Name:caf9248d57c69f6 MacAddress:42:58:3e:dd:c5:16 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:38:33:7c:81:03 Speed:0 Mtu:9001} {Name:f3a8e6fedb115f6 MacAddress:4e:68:7c:82:2f:49 Speed:10000 Mtu:8901} {Name:f5c643cc8380529 MacAddress:76:7d:51:fe:65:2e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:da:cc:be:17:f2:aa Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:4c:3b:71:c5:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.025586    4872 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.025623    4872 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.025881    4872 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.025903    4872 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-188.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026023    4872 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026031    4872 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026045    4872 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026063    4872 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026081    4872 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026214    4872 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026305    4872 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026459    4872 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026486    4872 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026502    4872 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026516    4872 kubelet.go:405] "Adding apiserver pod source"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.026526    4872 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.027578    4872 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.027593    4872 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:30:43.028241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.028182    4872 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 20:30:43.030984 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.030968    4872 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 20:30:43.030984 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.030982    4872 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:30:43.031089 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031079    4872 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 20:30:43.031262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031251    4872 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 20:30:43.031291 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031280    4872 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 20:30:43.031385 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031376    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 20:30:43.031415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031389    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 20:30:43.031415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031395    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 20:30:43.031415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031400    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 20:30:43.031415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031406    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 20:30:43.031415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031411    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 20:30:43.031415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031416    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031421    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031428    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031434    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031448    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031458    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031482    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 20:30:43.031587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.031487    4872 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 20:30:43.034633 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.034621    4872 server.go:1263] "Started kubelet"
Nov 01 20:30:43.035198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.034709    4872 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 20:30:43.035198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.034766    4872 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 20:30:43.035198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.034838    4872 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 20:30:43.035198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.035145    4872 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 20:30:43.035197 ip-10-0-131-188 systemd[1]: Started Kubernetes Kubelet.
Nov 01 20:30:43.036485 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.036189    4872 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 20:30:43.038635 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.038620    4872 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 20:30:43.038774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.038637    4872 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 20:30:43.038774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.038728    4872 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 20:30:43.038774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.038741    4872 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 20:30:43.038774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.038722    4872 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:03 +0000 UTC" deadline="2028-07-09 17:16:03.129922318 +0000 UTC"
Nov 01 20:30:43.038774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.038761    4872 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14780h45m20.091163867s"
Nov 01 20:30:43.039536 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.039512    4872 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-188.us-west-1.compute.internal\" not found"
Nov 01 20:30:43.043098 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043079    4872 factory.go:55] Registering systemd factory
Nov 01 20:30:43.043098 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043100    4872 factory.go:223] Registration of the systemd container factory successfully
Nov 01 20:30:43.043505 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043491    4872 factory.go:153] Registering CRI-O factory
Nov 01 20:30:43.043560 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043509    4872 factory.go:223] Registration of the crio container factory successfully
Nov 01 20:30:43.043606 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043566    4872 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 20:30:43.043606 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043585    4872 factory.go:103] Registering Raw factory
Nov 01 20:30:43.043606 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.043602    4872 manager.go:1199] Started watching for new ooms in manager
Nov 01 20:30:43.044052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.044038    4872 manager.go:319] Starting recovery of all containers
Nov 01 20:30:43.047013 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.046961    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6559ca8c-d167-4f33-95f9-1856858fc3d5" volumeName="kubernetes.io/projected/6559ca8c-d167-4f33-95f9-1856858fc3d5-kube-api-access-b5277" seLinuxMountContext=""
Nov 01 20:30:43.047096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047030    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83552290-2377-4a32-be4c-e1f10c121b12" volumeName="kubernetes.io/projected/83552290-2377-4a32-be4c-e1f10c121b12-kube-api-access-8bqc8" seLinuxMountContext=""
Nov 01 20:30:43.047096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047069    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" volumeName="kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-config" seLinuxMountContext=""
Nov 01 20:30:43.047096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047084    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="428f74c3-acde-4cbb-afd8-32c13b30cadf" volumeName="kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-tmp" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047104    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5912e397-27c0-48e4-a25d-a902436d452d" volumeName="kubernetes.io/secret/5912e397-27c0-48e4-a25d-a902436d452d-serving-cert" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047123    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" volumeName="kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047146    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" volumeName="kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-tmp" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047184    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b44fcb4-4d02-4b52-b0bb-b67dcaafd336" volumeName="kubernetes.io/projected/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336-kube-api-access-mf6wh" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047204    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b245f314-7357-4797-90d0-aa95f77b154a" volumeName="kubernetes.io/secret/b245f314-7357-4797-90d0-aa95f77b154a-serving-cert" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047233    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce97c7cb-9ccb-468e-af69-a8f8e922ec24" volumeName="kubernetes.io/configmap/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-serviceca" seLinuxMountContext=""
Nov 01 20:30:43.047262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047254    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc3e550-0cb5-4301-82bc-17503dc53deb" volumeName="kubernetes.io/secret/9cc3e550-0cb5-4301-82bc-17503dc53deb-serving-cert" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047266    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f5b52b-e610-4b2f-9521-aec1e69f2170" volumeName="kubernetes.io/projected/25f5b52b-e610-4b2f-9521-aec1e69f2170-kube-api-access-xhfqf" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047282    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047293    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5318ea66-6db7-4701-a20d-84b25e4c196c" volumeName="kubernetes.io/configmap/5318ea66-6db7-4701-a20d-84b25e4c196c-iptables-alerter-script" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047325    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" volumeName="kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-binary-copy" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047337    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26fed350-454f-4f57-8989-016d524f5b4f" volumeName="kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-multus-daemon-config" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047376    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32264030-2487-4a8d-b5e4-3f233831760b" volumeName="kubernetes.io/secret/32264030-2487-4a8d-b5e4-3f233831760b-agent-certs" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047405    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="428f74c3-acde-4cbb-afd8-32c13b30cadf" volumeName="kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-tuned" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047416    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" volumeName="kubernetes.io/projected/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-kube-api-access-fdhlw" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047432    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047457    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047474    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5318ea66-6db7-4701-a20d-84b25e4c196c" volumeName="kubernetes.io/projected/5318ea66-6db7-4701-a20d-84b25e4c196c-kube-api-access-zv9nk" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047513    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" volumeName="kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-service-ca-bundle" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047528    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" volumeName="kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-env-overrides" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047543    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" volumeName="kubernetes.io/secret/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047563    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b245f314-7357-4797-90d0-aa95f77b154a" volumeName="kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-trusted-ca" seLinuxMountContext=""
Nov 01 20:30:43.047586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047586    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b245f314-7357-4797-90d0-aa95f77b154a" volumeName="kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-config" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047600    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f5b52b-e610-4b2f-9521-aec1e69f2170" volumeName="kubernetes.io/configmap/25f5b52b-e610-4b2f-9521-aec1e69f2170-config-volume" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047674    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" volumeName="kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047698    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" volumeName="kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-snapshots" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047713    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260814c3-535b-448a-9213-2b992657efb8" volumeName="kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047726    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047759    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047801    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83552290-2377-4a32-be4c-e1f10c121b12" volumeName="kubernetes.io/configmap/83552290-2377-4a32-be4c-e1f10c121b12-telemetry-config" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047824    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f99c6ff8-8987-46e6-bfb3-8aea67369ea1" volumeName="kubernetes.io/projected/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-kube-api-access-zkxpg" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047842    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce97c7cb-9ccb-468e-af69-a8f8e922ec24" volumeName="kubernetes.io/projected/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-kube-api-access-zgdqs" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047853    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" volumeName="kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047868    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047894    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26fed350-454f-4f57-8989-016d524f5b4f" volumeName="kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-cni-binary-copy" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047907    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26fed350-454f-4f57-8989-016d524f5b4f" volumeName="kubernetes.io/projected/26fed350-454f-4f57-8989-016d524f5b4f-kube-api-access-lp849" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047931    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32264030-2487-4a8d-b5e4-3f233831760b" volumeName="kubernetes.io/configmap/32264030-2487-4a8d-b5e4-3f233831760b-konnectivity-ca" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047970    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd" volumeName="kubernetes.io/projected/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kube-api-access-q4lcr" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.047989    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048022    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ace44fda-7418-4639-83db-e7499a8e140d" volumeName="kubernetes.io/projected/ace44fda-7418-4639-83db-e7499a8e140d-kube-api-access-wlm2n" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048040    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b1356ae-ca5a-4464-8629-0272902c73cb" volumeName="kubernetes.io/projected/0b1356ae-ca5a-4464-8629-0272902c73cb-kube-api-access-mvrsg" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048060    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f5b52b-e610-4b2f-9521-aec1e69f2170" volumeName="kubernetes.io/empty-dir/25f5b52b-e610-4b2f-9521-aec1e69f2170-tmp-dir" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048082    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" volumeName="kubernetes.io/projected/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-kube-api-access-frfzs" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048100    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048121    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048149    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd018011-2b75-4090-812a-5b5039164aa9" volumeName="kubernetes.io/projected/bd018011-2b75-4090-812a-5b5039164aa9-kube-api-access-zzsrg" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048180    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10cbc89e-6b29-48e4-8218-03be6f3e3992" volumeName="kubernetes.io/empty-dir/10cbc89e-6b29-48e4-8218-03be6f3e3992-data-volume" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048192    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc3e550-0cb5-4301-82bc-17503dc53deb" volumeName="kubernetes.io/configmap/9cc3e550-0cb5-4301-82bc-17503dc53deb-config" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048216    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b20d35a5-f97b-48a6-9dfb-73c6d656032c" volumeName="kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-default-certificate" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048231    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b245f314-7357-4797-90d0-aa95f77b154a" volumeName="kubernetes.io/projected/b245f314-7357-4797-90d0-aa95f77b154a-kube-api-access-tqfgg" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048245    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10cbc89e-6b29-48e4-8218-03be6f3e3992" volumeName="kubernetes.io/projected/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-api-access-7j9xg" seLinuxMountContext=""
Nov 01 20:30:43.048285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048259    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048356    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ba0752-5a6d-4490-a0bc-e3819c68a504" volumeName="kubernetes.io/configmap/a0ba0752-5a6d-4490-a0bc-e3819c68a504-nginx-conf" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048383    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b20d35a5-f97b-48a6-9dfb-73c6d656032c" volumeName="kubernetes.io/projected/b20d35a5-f97b-48a6-9dfb-73c6d656032c-kube-api-access-j2m9k" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048409    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b20d35a5-f97b-48a6-9dfb-73c6d656032c" volumeName="kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-stats-auth" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048422    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd13ffc1-85a5-476f-83f8-3dac47e667a8" volumeName="kubernetes.io/projected/dd13ffc1-85a5-476f-83f8-3dac47e667a8-kube-api-access-xsfrt" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048440    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048453    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048475    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b26b214-b220-476f-af79-d819a9ba8c80" volumeName="kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048487    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" volumeName="kubernetes.io/secret/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-serving-cert" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048518    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b60f90ff-66ee-410f-b953-aee5ed9b8d47" volumeName="kubernetes.io/projected/b60f90ff-66ee-410f-b953-aee5ed9b8d47-kube-api-access-c7ks6" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048563    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="428f74c3-acde-4cbb-afd8-32c13b30cadf" volumeName="kubernetes.io/projected/428f74c3-acde-4cbb-afd8-32c13b30cadf-kube-api-access-wktzx" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048594    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc3e550-0cb5-4301-82bc-17503dc53deb" volumeName="kubernetes.io/projected/9cc3e550-0cb5-4301-82bc-17503dc53deb-kube-api-access-lwrkv" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048613    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c136bbc2-2ebc-40b7-8548-ca250b2dba7c" volumeName="kubernetes.io/configmap/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-config" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048633    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c136bbc2-2ebc-40b7-8548-ca250b2dba7c" volumeName="kubernetes.io/projected/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-kube-api-access-rgd6z" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048644    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5912e397-27c0-48e4-a25d-a902436d452d" volumeName="kubernetes.io/configmap/5912e397-27c0-48e4-a25d-a902436d452d-config" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048692    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cdfd004-c495-4f69-9aba-35276b089b06" volumeName="kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048704    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6559ca8c-d167-4f33-95f9-1856858fc3d5" volumeName="kubernetes.io/empty-dir/6559ca8c-d167-4f33-95f9-1856858fc3d5-tmp-dir" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048728    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" volumeName="kubernetes.io/projected/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-kube-api-access-4x8wh" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048750    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ce2d60-a95b-4a6a-b793-1577fbb86703" volumeName="kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048775    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5912e397-27c0-48e4-a25d-a902436d452d" volumeName="kubernetes.io/projected/5912e397-27c0-48e4-a25d-a902436d452d-kube-api-access-fxhdg" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048791    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" volumeName="kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048812    4872 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b60f90ff-66ee-410f-b953-aee5ed9b8d47" volumeName="kubernetes.io/secret/b60f90ff-66ee-410f-b953-aee5ed9b8d47-cert" seLinuxMountContext=""
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048832    4872 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.048839    4872 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 20:30:43.049666 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.049003    4872 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 20:30:43.062967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.062945    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/1.log"
Nov 01 20:30:43.063499 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.063481    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/1.log"
Nov 01 20:30:43.069063 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.069026    4872 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 20:30:43.070941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.070887    4872 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 20:30:43.070941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.070940    4872 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 20:30:43.071067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.070964    4872 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 20:30:43.071067 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.071017    4872 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 20:30:43.077410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.077393    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:43.077502 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.077456    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:43.099614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.099557    4872 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="b97cddfaa7af2f00f54ba7c3e5f599ccbe8bcdca8875f7837079ec0c9ea3513d" exitCode=0
Nov 01 20:30:43.099614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.099583    4872 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="45f81759989cfcb16d839265c09f03490955911e1fd6e895001de63e9d6bfe5d" exitCode=0
Nov 01 20:30:43.099614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.099592    4872 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="0f375da57225f491fb201d1b7d01a2de801f25340f3bf56df9ce8743bff23fa1" exitCode=0
Nov 01 20:30:43.099614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.099602    4872 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="3ff2978bb97dc9bea40993aabb403f65e252a9421c05b2bf89bc54c5637193cd" exitCode=0
Nov 01 20:30:43.099614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.099610    4872 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="e40b7a01122f01b8b2afb57eadc09cf9a6b830702b6412a77a290f1d10caf7c2" exitCode=0
Nov 01 20:30:43.099861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.099618    4872 generic.go:356] "Generic (PLEG): container finished" podID="6fe4d6b3-46b7-4c61-88e0-a130c95e31d5" containerID="01c9d88cbb0bb83f1f29b43fbef4edfe531b5f4807089608dc7ced9476b62700" exitCode=0
Nov 01 20:30:43.102856 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.102835    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/1.log"
Nov 01 20:30:43.102938 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.102870    4872 generic.go:356] "Generic (PLEG): container finished" podID="b245f314-7357-4797-90d0-aa95f77b154a" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0" exitCode=255
Nov 01 20:30:43.112208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.112186    4872 generic.go:356] "Generic (PLEG): container finished" podID="905080b6348e87e21e0923f43ca1286f" containerID="a89aeefefd04a0b64514117dc3a5b065ca0b0e32d798602f21f31f345ae42141" exitCode=0
Nov 01 20:30:43.124422 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.124404    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:30:43.124725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.124706    4872 generic.go:356] "Generic (PLEG): container finished" podID="9e3961b5-f2f0-4c47-9017-72de63a2f7c4" containerID="576bef084d5d240d58cf1b55d2e0276e0b70ef07ccb7b987f27675a0e263f4e9" exitCode=1
Nov 01 20:30:43.137519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.137503    4872 manager.go:324] Recovery completed
Nov 01 20:30:43.139579 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.139554    4872 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-188.us-west-1.compute.internal\" not found"
Nov 01 20:30:43.151908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.151893    4872 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:43.152946 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.152925    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:43.153000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.152952    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:43.153000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.152962    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:43.154771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154757    4872 cpu_manager.go:226] "Starting" policy="none"
Nov 01 20:30:43.154771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154768    4872 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 20:30:43.154901 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154775    4872 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:43.154901 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154783    4872 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 20:30:43.154901 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154881    4872 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 20:30:43.154901 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154891    4872 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 20:30:43.155029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154907    4872 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 20:30:43.155029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154912    4872 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 20:30:43.155029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154917    4872 policy_none.go:50] "Start"
Nov 01 20:30:43.155029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154924    4872 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 20:30:43.155029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.154932    4872 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 20:30:43.155536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.155523    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:43.155590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.155547    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:43.155590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.155562    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:43.163221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.163205    4872 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 20:30:43.163221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.163221    4872 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 20:30:43.163307 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.163229    4872 policy_none.go:44] "Start"
Nov 01 20:30:43.169061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.169035    4872 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 20:30:43.171798 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.171775    4872 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 20:30:43.172403 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172390    4872 manager.go:341] "Starting Device Plugin manager"
Nov 01 20:30:43.172448 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.172416    4872 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 20:30:43.172448 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172424    4872 server.go:87] "Starting device plugin registration server"
Nov 01 20:30:43.172744 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172731    4872 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 20:30:43.172793 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172747    4872 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 20:30:43.172882 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172869    4872 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 20:30:43.172992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172951    4872 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 20:30:43.172992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172988    4872 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 20:30:43.173105 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.172995    4872 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 20:30:43.173159 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.173057    4872 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:30:43.172978654Z","UUID":"8068325f-5186-478d-91b3-c1aead461a3b","Handler":null,"Name":"","Endpoint":""}
Nov 01 20:30:43.174263 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.174247    4872 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 20:30:43.174352 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.174267    4872 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 20:30:43.174746 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.174716    4872 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 20:30:43.174843 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.174757    4872 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-188.us-west-1.compute.internal\" not found"
Nov 01 20:30:43.239000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.238982    4872 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:43.239104 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.239080    4872 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 20:30:43.239401 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.239387    4872 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:43.239482 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.239404    4872 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:43.273927 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.273907    4872 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:30:43.274716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.274701    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:30:43.274792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.274726    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:30:43.274792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.274735    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:30:43.274792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.274768    4872 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.290209 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.290189    4872 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:30:43.372018 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.371941    4872 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"]
Nov 01 20:30:43.372520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372488    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"434dabbcb9fcd49f00b858143ea6ae44","Type":"ContainerStarted","Data":"b6ab0c2705e17a836ebb3c38c826150ef861ef317fbf84c817ca271e7bf74a87"}
Nov 01 20:30:43.372598 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372529    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"434dabbcb9fcd49f00b858143ea6ae44","Type":"ContainerStarted","Data":"fbcc760508b3838a4e4aa596158ddc498c1a78e73241101888f496c74638849e"}
Nov 01 20:30:43.372650 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372600    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"905080b6348e87e21e0923f43ca1286f","Type":"ContainerStarted","Data":"ce1338c36a2a5f12d83fa771fecf576ef5d1618c79b4cfa16c5669799688b3a8"}
Nov 01 20:30:43.372650 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372622    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"905080b6348e87e21e0923f43ca1286f","Type":"ContainerDied","Data":"a89aeefefd04a0b64514117dc3a5b065ca0b0e32d798602f21f31f345ae42141"}
Nov 01 20:30:43.372650 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372643    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal" event={"ID":"905080b6348e87e21e0923f43ca1286f","Type":"ContainerStarted","Data":"56599c60f7e453f81cf8a9f508e6b0c063492fdf92520a76d0b4172c8fa3e977"}
Nov 01 20:30:43.372792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372689    4872 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.372792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.372689    4872 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.384048 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.384030    4872 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.384145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.384084    4872 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.384480 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.384466    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeNotReady"
Nov 01 20:30:43.384530 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.384492    4872 setters.go:546] "Node became not ready" node="ip-10-0-131-188.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T20:30:43Z","lastTransitionTime":"2026-11-01T20:30:43Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 20:30:43.400732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.400709    4872 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-188.us-west-1.compute.internal" is owned by different node (oldNodeID="e1d90138-7d47-4668-8c66-5dd23f2de26c", newNodeID=""), cleaning up...
Nov 01 20:30:43.435052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.435032    4872 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:30:43.435110 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.435078    4872 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.435453 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.435439    4872 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:30:43.435491 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:43.435473    4872 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.541526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.541500    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/434dabbcb9fcd49f00b858143ea6ae44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"434dabbcb9fcd49f00b858143ea6ae44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.541615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.541533    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.541615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.541577    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.642471 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.642410    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/434dabbcb9fcd49f00b858143ea6ae44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"434dabbcb9fcd49f00b858143ea6ae44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.642471 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.642451    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.642471 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.642468    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.642611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.642483    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/434dabbcb9fcd49f00b858143ea6ae44-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"434dabbcb9fcd49f00b858143ea6ae44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.642611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.642503    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:43.642611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:43.642507    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/905080b6348e87e21e0923f43ca1286f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" (UID: \"905080b6348e87e21e0923f43ca1286f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:44.028740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.028719    4872 apiserver.go:52] "Watching apiserver"
Nov 01 20:30:44.127516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.127494    4872 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:44.127623 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.127613    4872 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:44.185576 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.185554    4872 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 20:30:44.186200 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.186180    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-92gcr","kube-system/konnectivity-agent-f7pr8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-l7c7b","openshift-ovn-kubernetes/ovnkube-node-phs9n","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5","openshift-image-registry/image-registry-54cb57cf74-86lck","openshift-dns/dns-default-glm2w","openshift-image-registry/node-ca-w2766","openshift-multus/multus-additional-cni-plugins-fspt2","openshift-network-diagnostics/network-check-target-j4t6k","openshift-console-operator/console-operator-7cd56b8dbc-9m5fw","openshift-ingress-canary/ingress-canary-dwl2n","openshift-image-registry/image-registry-6b48f6559f-rxmwf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t","kube-system/kas-connection-checker-685895c7c9-75vgh","openshift-cluster-node-tuning-operator/tuned-xbb76","openshift-ingress/router-default-66ff46cf4c-rt6qk","openshift-insights/insights-runtime-extractor-ntncn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn","openshift-multus/multus-t2v7x","openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k","kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k","openshift-insights/insights-operator-7c4986d449-v9rgh","openshift-network-console/networking-console-plugin-86c478bb88-sr6nr","kube-system/global-pull-secret-syncer-8x2dv","kube-system/kas-connection-checker-685895c7c9-76pfm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq","openshift-multus/network-metrics-daemon-cx7ds","openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf","kube-system/kas-connection-checker-685895c7c9-pjl9s"]
Nov 01 20:30:44.188080 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.188064    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.188342 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.188326    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.189419 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.189404    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.190147 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.190126    4872 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:30:44.190216 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.190190    4872 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:44.190334 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.190321    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.190554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.190533    4872 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:30:44.190629 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.190581    4872 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.us-west-1.compute.internal"
Nov 01 20:30:44.191131 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.191116    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:44.197047 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.197030    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:44.198885 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.198867    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.199726 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.199712    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.200257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.200240    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.241340 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.241311    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.241415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.241349    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.242378 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242359    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.242440 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242401    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.242440 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242432    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 20:30:44.242507 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242446    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 20:30:44.242507 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242467    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 20:30:44.242617 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242603    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.242651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242631    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9m2k9\""
Nov 01 20:30:44.242745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.242731    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.243090 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.243076    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gnklv\""
Nov 01 20:30:44.243144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.243088    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.243144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.243114    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.243329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.243315    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xct89\""
Nov 01 20:30:44.243572 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.243559    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 20:30:44.243861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.243847    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 20:30:44.251512 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.251495    4872 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 20:30:44.340315 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340265    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.340398 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340371    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.340452 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340433    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.340495 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340472    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.340686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340645    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.340766 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340749    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 20:30:44.340823 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340779    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 20:30:44.340864 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340837    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.340898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.340872    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.341167 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341148    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.341223 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341189    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.341315 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341297    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 20:30:44.341373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341299    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.341373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341341    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 20:30:44.341373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341349    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.341597 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341581    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.341750 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341620    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 20:30:44.341750 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341673    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.341750 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341708    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 20:30:44.341851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341736    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.341851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341766    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 20:30:44.341988 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.341974    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.342102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342086    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 20:30:44.342164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342088    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 20:30:44.342164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342103    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.342164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342105    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 20:30:44.342279 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342265    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 20:30:44.342395 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342384    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 20:30:44.342451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342412    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 20:30:44.342590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342573    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.342639 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342580    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 20:30:44.342702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342675    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.342809 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342795    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 20:30:44.342898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342867    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.342898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342802    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.342898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342809    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 20:30:44.342898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342840    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342861    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342823    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342956    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342974    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342996    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343003    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342988    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9xjd\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.342995    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343036    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343050    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cpf8s\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343065    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 20:30:44.343151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343075    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jgj4\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343221    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343237    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343222    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343238    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343265    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343287    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343291    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343288    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343311    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343339    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343347    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2nmd8\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343355    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pklm5\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343396    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kzbx4\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343347    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343409    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343414    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343441    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pjdf7\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343446    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9chrm\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343449    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343481    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343497    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8z8c\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343509    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bdld\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343434    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343501    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wk8gn\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343535    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-njcwg\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343546    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pskvw\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343549    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 20:30:44.343676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343599    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n2j5c\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343609    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xls9n\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343644    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h288r\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.343955    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344115    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344193    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ggf7k\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344254    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-pmwqj\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344273    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cg54x\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344444    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5cgzw\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344482    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zmb7f\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344563    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r4cnd\""
Nov 01 20:30:44.344853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.344683    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j72sw\""
Nov 01 20:30:44.345797 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.345768    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.345923 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.345832    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336-kube-api-access-mf6wh\") pod \"kas-connection-checker-685895c7c9-76pfm\" (UID: \"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336\") " pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:44.345923 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.345885    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.346067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.345919    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ks6\" (UniqueName: \"kubernetes.io/projected/b60f90ff-66ee-410f-b953-aee5ed9b8d47-kube-api-access-c7ks6\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:44.346067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.345982    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-etc-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.346067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.346008    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-snapshots\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.346230 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.346112    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-snapshots\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.346282 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.346234    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.346362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.346340    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.346726 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.346705    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.349345 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349325    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2m9k\" (UniqueName: \"kubernetes.io/projected/b20d35a5-f97b-48a6-9dfb-73c6d656032c-kube-api-access-j2m9k\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.349430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349362    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.349430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349384    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfrt\" (UniqueName: \"kubernetes.io/projected/dd13ffc1-85a5-476f-83f8-3dac47e667a8-kube-api-access-xsfrt\") pod \"network-check-source-6f78d59f99-jm5qf\" (UID: \"dd13ffc1-85a5-476f-83f8-3dac47e667a8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:44.349430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349408    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.349579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349456    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8wh\" (UniqueName: \"kubernetes.io/projected/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-kube-api-access-4x8wh\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.349579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349493    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.349579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349533    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfzs\" (UniqueName: \"kubernetes.io/projected/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-kube-api-access-frfzs\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.349579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349571    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-config\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.349894 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349873    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.349956 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349946    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-config\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.350023 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.349995    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-socket-dir-parent\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.350076 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350011    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.350128 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350089    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-multus-certs\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.350808 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350784    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhfqf\" (UniqueName: \"kubernetes.io/projected/25f5b52b-e610-4b2f-9521-aec1e69f2170-kube-api-access-xhfqf\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.350881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350828    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.350881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350861    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:44.350969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350887    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cnibin\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.350969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350911    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.350969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350939    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.351093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.350965    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.351093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351024    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-multus-daemon-config\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.351093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351061    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.351232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351089    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhlw\" (UniqueName: \"kubernetes.io/projected/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-kube-api-access-fdhlw\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.351232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351119    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.351232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351223    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-sys-fs\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.351362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351250    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32264030-2487-4a8d-b5e4-3f233831760b-konnectivity-ca\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:44.351362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351260    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.351362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351274    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhdg\" (UniqueName: \"kubernetes.io/projected/5912e397-27c0-48e4-a25d-a902436d452d-kube-api-access-fxhdg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:44.351362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351216    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.351362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351299    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351404    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351444    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b245f314-7357-4797-90d0-aa95f77b154a-serving-cert\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351450    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-multus-daemon-config\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351463    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351470    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-kubernetes\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351479    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351505    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-dbus\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351525    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10cbc89e-6b29-48e4-8218-03be6f3e3992-crio-socket\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351532    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32264030-2487-4a8d-b5e4-3f233831760b-konnectivity-ca\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351547    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0ba0752-5a6d-4490-a0bc-e3819c68a504-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:44.351586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351582    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6559ca8c-d167-4f33-95f9-1856858fc3d5-hosts-file\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351638    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b245f314-7357-4797-90d0-aa95f77b154a-serving-cert\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351690    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5277\" (UniqueName: \"kubernetes.io/projected/6559ca8c-d167-4f33-95f9-1856858fc3d5-kube-api-access-b5277\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351717    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9xg\" (UniqueName: \"kubernetes.io/projected/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-api-access-7j9xg\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351759    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0ba0752-5a6d-4490-a0bc-e3819c68a504-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351763    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-sys\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351793    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351817    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351834    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351861    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-modprobe-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351943    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysconfig\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.351985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.351969    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6559ca8c-d167-4f33-95f9-1856858fc3d5-tmp-dir\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352024    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6559ca8c-d167-4f33-95f9-1856858fc3d5-tmp-dir\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352022    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352031    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cc3e550-0cb5-4301-82bc-17503dc53deb-serving-cert\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352063    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd6z\" (UniqueName: \"kubernetes.io/projected/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-kube-api-access-rgd6z\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352078    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-conf\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352108    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp849\" (UniqueName: \"kubernetes.io/projected/26fed350-454f-4f57-8989-016d524f5b4f-kube-api-access-lp849\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352126    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5912e397-27c0-48e4-a25d-a902436d452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352163    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cc3e550-0cb5-4301-82bc-17503dc53deb-serving-cert\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352164    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-ovn\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352229    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsrg\" (UniqueName: \"kubernetes.io/projected/bd018011-2b75-4090-812a-5b5039164aa9-kube-api-access-zzsrg\") pod \"kas-connection-checker-685895c7c9-75vgh\" (UID: \"bd018011-2b75-4090-812a-5b5039164aa9\") " pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:44.352265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352254    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-bin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352290    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5912e397-27c0-48e4-a25d-a902436d452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352345    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-host\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352368    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-script-lib\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352397    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f5b52b-e610-4b2f-9521-aec1e69f2170-tmp-dir\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352413    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqc8\" (UniqueName: \"kubernetes.io/projected/83552290-2377-4a32-be4c-e1f10c121b12-kube-api-access-8bqc8\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352427    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352458    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrkv\" (UniqueName: \"kubernetes.io/projected/9cc3e550-0cb5-4301-82bc-17503dc53deb-kube-api-access-lwrkv\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352478    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f5b52b-e610-4b2f-9521-aec1e69f2170-tmp-dir\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352496    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-systemd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352525    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-config\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352565    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.352591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352585    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-kubelet\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352606    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5912e397-27c0-48e4-a25d-a902436d452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352631    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-log-socket\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352674    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352787    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlm2n\" (UniqueName: \"kubernetes.io/projected/ace44fda-7418-4639-83db-e7499a8e140d-kube-api-access-wlm2n\") pod \"kas-connection-checker-685895c7c9-pjl9s\" (UID: \"ace44fda-7418-4639-83db-e7499a8e140d\") " pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352812    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352842    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-config\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352844    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-device-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352906    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovnkube-script-lib\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352954    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352975    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5912e397-27c0-48e4-a25d-a902436d452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.352992    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32264030-2487-4a8d-b5e4-3f233831760b-agent-certs\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353034    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-system-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353065    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-serviceca\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.353118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353093    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-run\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353132    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-cnibin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353218    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9cc3e550-0cb5-4301-82bc-17503dc53deb-config\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353245    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353249    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32264030-2487-4a8d-b5e4-3f233831760b-agent-certs\") pod \"konnectivity-agent-f7pr8\" (UID: \"32264030-2487-4a8d-b5e4-3f233831760b\") " pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353254    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-serviceca\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353266    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353347    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353391    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353415    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9cc3e550-0cb5-4301-82bc-17503dc53deb-config\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353418    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-netns\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353450    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-hostroot\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353474    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-systemd-units\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.353520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353504    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60f90ff-66ee-410f-b953-aee5ed9b8d47-cert\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353529    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktzx\" (UniqueName: \"kubernetes.io/projected/428f74c3-acde-4cbb-afd8-32c13b30cadf-kube-api-access-wktzx\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353558    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-k8s-cni-cncf-io\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353561    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfd004-c495-4f69-9aba-35276b089b06-original-pull-secret\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353679    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-var-lib-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353703    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353708    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60f90ff-66ee-410f-b953-aee5ed9b8d47-cert\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353736    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdqs\" (UniqueName: \"kubernetes.io/projected/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-kube-api-access-zgdqs\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353808    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.353880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353818    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-netns\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.354113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353955    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10cbc89e-6b29-48e4-8218-03be6f3e3992-data-volume\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.354113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.353988    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.354113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354006    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.354113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354047    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-stats-auth\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.354113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354060    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10cbc89e-6b29-48e4-8218-03be6f3e3992-data-volume\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.354113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354081    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-systemd\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.354294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354124    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.354294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354208    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-tuned\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.354294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354231    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-etc-kubernetes\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.354294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354243    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-stats-auth\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.354294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354254    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5318ea66-6db7-4701-a20d-84b25e4c196c-iptables-alerter-script\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.354294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354277    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9nk\" (UniqueName: \"kubernetes.io/projected/5318ea66-6db7-4701-a20d-84b25e4c196c-kube-api-access-zv9nk\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354323    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-tuned\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354327    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354363    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354391    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfgg\" (UniqueName: \"kubernetes.io/projected/b245f314-7357-4797-90d0-aa95f77b154a-kube-api-access-tqfgg\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354417    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5318ea66-6db7-4701-a20d-84b25e4c196c-host-slash\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354422    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.354567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354466    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354583    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5318ea66-6db7-4701-a20d-84b25e4c196c-iptables-alerter-script\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354601    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-node-log\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354609    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354649    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354705    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354732    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354765    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.354906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354888    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-system-cni-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354911    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354921    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-host\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354950    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lcr\" (UniqueName: \"kubernetes.io/projected/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kube-api-access-q4lcr\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.354976    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-conf-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355020    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-netd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355046    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-slash\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355073    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-tmp\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355115    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxpg\" (UniqueName: \"kubernetes.io/projected/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-kube-api-access-zkxpg\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:44.355139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355131    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-multus\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355149    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-registration-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355165    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/83552290-2377-4a32-be4c-e1f10c121b12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355180    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-lib-modules\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355192    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-tmp\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355219    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-var-lib-kubelet\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355235    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrsg\" (UniqueName: \"kubernetes.io/projected/0b1356ae-ca5a-4464-8629-0272902c73cb-kube-api-access-mvrsg\") pod \"volume-data-source-validator-58c9f5f46d-rs7t5\" (UID: \"0b1356ae-ca5a-4464-8629-0272902c73cb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355251    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-serving-cert\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355270    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355286    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355301    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355348    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-trusted-ca\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355362    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-cni-binary-copy\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.355377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355377    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355393    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355413    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-default-certificate\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355428    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-tmp\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355447    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-kubelet\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355460    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-bin\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355474    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-socket-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355491    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-kubelet-config\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355506    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-env-overrides\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.355716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355580    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5b52b-e610-4b2f-9521-aec1e69f2170-config-volume\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.355982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355774    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25f5b52b-e610-4b2f-9521-aec1e69f2170-config-volume\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.355982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.355863    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-env-overrides\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.356033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.356002    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/83552290-2377-4a32-be4c-e1f10c121b12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.356343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.356319    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-serving-cert\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.356610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.356596    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26fed350-454f-4f57-8989-016d524f5b4f-cni-binary-copy\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.356726 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.356708    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.356771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.356733    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/428f74c3-acde-4cbb-afd8-32c13b30cadf-tmp\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.356891 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.356877    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-default-certificate\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.391907 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.391880    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 20:30:44.391979 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.391949    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 20:30:44.393503 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.393469    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.397377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.397361    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b245f314-7357-4797-90d0-aa95f77b154a-trusted-ca\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.397430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.397416    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.404144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.404128    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/9b44fcb4-4d02-4b52-b0bb-b67dcaafd336-kube-api-access-mf6wh\") pod \"kas-connection-checker-685895c7c9-76pfm\" (UID: \"9b44fcb4-4d02-4b52-b0bb-b67dcaafd336\") " pod="kube-system/kas-connection-checker-685895c7c9-76pfm"
Nov 01 20:30:44.404272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.404256    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ks6\" (UniqueName: \"kubernetes.io/projected/b60f90ff-66ee-410f-b953-aee5ed9b8d47-kube-api-access-c7ks6\") pod \"ingress-canary-dwl2n\" (UID: \"b60f90ff-66ee-410f-b953-aee5ed9b8d47\") " pod="openshift-ingress-canary/ingress-canary-dwl2n"
Nov 01 20:30:44.405312 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.405297    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2m9k\" (UniqueName: \"kubernetes.io/projected/b20d35a5-f97b-48a6-9dfb-73c6d656032c-kube-api-access-j2m9k\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.406156 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.406142    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8wh\" (UniqueName: \"kubernetes.io/projected/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-kube-api-access-4x8wh\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.406344 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.406330    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfrt\" (UniqueName: \"kubernetes.io/projected/dd13ffc1-85a5-476f-83f8-3dac47e667a8-kube-api-access-xsfrt\") pod \"network-check-source-6f78d59f99-jm5qf\" (UID: \"dd13ffc1-85a5-476f-83f8-3dac47e667a8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jm5qf"
Nov 01 20:30:44.408150 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.408132    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.410090 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410075    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhlw\" (UniqueName: \"kubernetes.io/projected/8b5d9c2b-5435-40c7-bafb-a6b1b741e37b-kube-api-access-fdhlw\") pod \"insights-operator-7c4986d449-v9rgh\" (UID: \"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b\") " pod="openshift-insights/insights-operator-7c4986d449-v9rgh"
Nov 01 20:30:44.410936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410608    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9xg\" (UniqueName: \"kubernetes.io/projected/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-api-access-7j9xg\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.410936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410622    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrkv\" (UniqueName: \"kubernetes.io/projected/9cc3e550-0cb5-4301-82bc-17503dc53deb-kube-api-access-lwrkv\") pod \"service-ca-operator-544d749c8c-5476k\" (UID: \"9cc3e550-0cb5-4301-82bc-17503dc53deb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k"
Nov 01 20:30:44.410936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410786    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsrg\" (UniqueName: \"kubernetes.io/projected/bd018011-2b75-4090-812a-5b5039164aa9-kube-api-access-zzsrg\") pod \"kas-connection-checker-685895c7c9-75vgh\" (UID: \"bd018011-2b75-4090-812a-5b5039164aa9\") " pod="kube-system/kas-connection-checker-685895c7c9-75vgh"
Nov 01 20:30:44.410936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410897    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5277\" (UniqueName: \"kubernetes.io/projected/6559ca8c-d167-4f33-95f9-1856858fc3d5-kube-api-access-b5277\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.411167 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410938    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfzs\" (UniqueName: \"kubernetes.io/projected/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-kube-api-access-frfzs\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.411167 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.410981    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqc8\" (UniqueName: \"kubernetes.io/projected/83552290-2377-4a32-be4c-e1f10c121b12-kube-api-access-8bqc8\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.411167 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411032    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxmw\" (UniqueName: \"kubernetes.io/projected/260814c3-535b-448a-9213-2b992657efb8-kube-api-access-lxxmw\") pod \"network-check-target-j4t6k\" (UID: \"260814c3-535b-448a-9213-2b992657efb8\") " pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:44.411276 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411176    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd6z\" (UniqueName: \"kubernetes.io/projected/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-kube-api-access-rgd6z\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.411276 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411180    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp849\" (UniqueName: \"kubernetes.io/projected/26fed350-454f-4f57-8989-016d524f5b4f-kube-api-access-lp849\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.411276 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411257    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhfqf\" (UniqueName: \"kubernetes.io/projected/25f5b52b-e610-4b2f-9521-aec1e69f2170-kube-api-access-xhfqf\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.411521 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411500    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhdg\" (UniqueName: \"kubernetes.io/projected/5912e397-27c0-48e4-a25d-a902436d452d-kube-api-access-fxhdg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn\" (UID: \"5912e397-27c0-48e4-a25d-a902436d452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn"
Nov 01 20:30:44.411749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411735    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9nk\" (UniqueName: \"kubernetes.io/projected/5318ea66-6db7-4701-a20d-84b25e4c196c-kube-api-access-zv9nk\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.411749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.411744    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlm2n\" (UniqueName: \"kubernetes.io/projected/ace44fda-7418-4639-83db-e7499a8e140d-kube-api-access-wlm2n\") pod \"kas-connection-checker-685895c7c9-pjl9s\" (UID: \"ace44fda-7418-4639-83db-e7499a8e140d\") " pod="kube-system/kas-connection-checker-685895c7c9-pjl9s"
Nov 01 20:30:44.413027 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.413010    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdqs\" (UniqueName: \"kubernetes.io/projected/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-kube-api-access-zgdqs\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.413643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.413628    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.414075 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.414056    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lcr\" (UniqueName: \"kubernetes.io/projected/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kube-api-access-q4lcr\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.414113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.414074    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.414270 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.414250    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfgg\" (UniqueName: \"kubernetes.io/projected/b245f314-7357-4797-90d0-aa95f77b154a-kube-api-access-tqfgg\") pod \"console-operator-7cd56b8dbc-9m5fw\" (UID: \"b245f314-7357-4797-90d0-aa95f77b154a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:44.414551 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.414526    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktzx\" (UniqueName: \"kubernetes.io/projected/428f74c3-acde-4cbb-afd8-32c13b30cadf-kube-api-access-wktzx\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.414614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.414606    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.414650 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.414604    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxpg\" (UniqueName: \"kubernetes.io/projected/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-kube-api-access-zkxpg\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:44.415430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.415414    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrsg\" (UniqueName: \"kubernetes.io/projected/0b1356ae-ca5a-4464-8629-0272902c73cb-kube-api-access-mvrsg\") pod \"volume-data-source-validator-58c9f5f46d-rs7t5\" (UID: \"0b1356ae-ca5a-4464-8629-0272902c73cb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rs7t5"
Nov 01 20:30:44.455990 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.455962    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-conf\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.456072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.455997    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-ovn\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456021    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-bin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456042    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-host\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.456072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456066    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456069    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-ovn\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456085    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-bin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456093    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-systemd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456112    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce97c7cb-9ccb-468e-af69-a8f8e922ec24-host\") pod \"node-ca-w2766\" (UID: \"ce97c7cb-9ccb-468e-af69-a8f8e922ec24\") " pod="openshift-image-registry/node-ca-w2766"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456129    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-kubelet\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456135    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-conf\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456157    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-log-socket\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456174    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456181    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-device-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456132    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-systemd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456218    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456224    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-device-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.456239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456227    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-kubelet\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456244    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-system-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456267    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-run\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456271    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456235    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-log-socket\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456284    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-system-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456289    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-cnibin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456316    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-run\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456328    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456351    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-cnibin\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456374    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456395    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456445    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-netns\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456449    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456463    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-hostroot\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456478    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-systemd-units\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456493    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-k8s-cni-cncf-io\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456453    4872 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456512    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-var-lib-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456522    4872 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456539    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-systemd-units\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456546    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-k8s-cni-cncf-io\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456556    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-var-lib-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456455    4872 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456570    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-netns\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456575    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-netns\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456580    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-hostroot\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456592    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.956573061 +0000 UTC m=+2.005682872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456623    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.95660912 +0000 UTC m=+2.005718930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : secret "metrics-daemon-secret" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456630    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-netns\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456640    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456687    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-systemd\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456713    4872 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456728    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-etc-kubernetes\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456748    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.956736492 +0000 UTC m=+2.005846302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : configmap "kube-rbac-proxy" not found
Nov 01 20:30:44.456732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456751    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-systemd\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456761    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-etc-kubernetes\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456771    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5318ea66-6db7-4701-a20d-84b25e4c196c-host-slash\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456786    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-node-log\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456806    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456834    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456855    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-node-log\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456859    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-system-cni-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456836    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5318ea66-6db7-4701-a20d-84b25e4c196c-host-slash\") pod \"iptables-alerter-l7c7b\" (UID: \"5318ea66-6db7-4701-a20d-84b25e4c196c\") " pod="openshift-network-operator/iptables-alerter-l7c7b"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456870    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456886    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-host\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456896    4872 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456905    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-conf-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.456933    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.956921515 +0000 UTC m=+2.006031328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456923    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-netd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456954    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-netd\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456937    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-host\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456967    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-conf-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456898    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-system-cni-dir\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456962    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-slash\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.456991    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-slash\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457004    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-multus\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457022    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-registration-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457043    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-lib-modules\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457059    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-var-lib-kubelet\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457076    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457097    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457118    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-kubelet\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457137    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-bin\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457136    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-var-lib-kubelet\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457168    4872 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457171    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-kubelet\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457188    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-lib-modules\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457192    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-registration-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457202    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-cni-bin\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457195    4872 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457221    4872 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457206    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-var-lib-cni-multus\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457208    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.957194703 +0000 UTC m=+2.006304519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457272    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-socket-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457287    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-kubelet-config\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457307    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457316    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-socket-dir\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457344    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.95732944 +0000 UTC m=+2.006439243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457355    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-kubelet-config\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457366    4872 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457408    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.957396122 +0000 UTC m=+2.006505937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457420    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-etc-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457438    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457454    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457471    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-socket-dir-parent\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457487    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-multus-certs\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457492    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-etc-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457512    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cnibin\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457534    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-host-run-multus-certs\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457538    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-socket-dir-parent\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457536    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457566    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-run-openvswitch\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457572    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457578    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysctl-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457572    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe4d6b3-46b7-4c61-88e0-a130c95e31d5-cnibin\") pod \"multus-additional-cni-plugins-fspt2\" (UID: \"6fe4d6b3-46b7-4c61-88e0-a130c95e31d5\") " pod="openshift-multus/multus-additional-cni-plugins-fspt2"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457542    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26fed350-454f-4f57-8989-016d524f5b4f-multus-cni-dir\") pod \"multus-t2v7x\" (UID: \"26fed350-454f-4f57-8989-016d524f5b4f\") " pod="openshift-multus/multus-t2v7x"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457599    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457616    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-sys-fs\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457632    4872 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457637    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457667    4872 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457686    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-kubernetes\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457695    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.957680807 +0000 UTC m=+2.006790604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457701    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd-sys-fs\") pod \"aws-ebs-csi-driver-node-jnqsq\" (UID: \"77ec5f83-b7c3-4e1b-aa70-5a3d3da96afd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnqsq"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457715    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e3961b5-f2f0-4c47-9017-72de63a2f7c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9n\" (UID: \"9e3961b5-f2f0-4c47-9017-72de63a2f7c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457732    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.957719241 +0000 UTC m=+2.006829047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457731    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-kubernetes\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457750    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-dbus\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457770    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10cbc89e-6b29-48e4-8218-03be6f3e3992-crio-socket\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457778    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfd004-c495-4f69-9aba-35276b089b06-dbus\") pod \"global-pull-secret-syncer-8x2dv\" (UID: \"1cdfd004-c495-4f69-9aba-35276b089b06\") " pod="kube-system/global-pull-secret-syncer-8x2dv"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457785    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6559ca8c-d167-4f33-95f9-1856858fc3d5-hosts-file\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457800    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-sys\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457813    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10cbc89e-6b29-48e4-8218-03be6f3e3992-crio-socket\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457815    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457841    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6559ca8c-d167-4f33-95f9-1856858fc3d5-hosts-file\") pod \"node-resolver-92gcr\" (UID: \"6559ca8c-d167-4f33-95f9-1856858fc3d5\") " pod="openshift-dns/node-resolver-92gcr"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457841    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457864    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-sys\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457879    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-modprobe-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457885    4872 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457904    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysconfig\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.457924 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457908    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.957899519 +0000 UTC m=+2.007009320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:44.460126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457945    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-modprobe-d\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.460126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.457948    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/428f74c3-acde-4cbb-afd8-32c13b30cadf-etc-sysconfig\") pod \"tuned-xbb76\" (UID: \"428f74c3-acde-4cbb-afd8-32c13b30cadf\") " pod="openshift-cluster-node-tuning-operator/tuned-xbb76"
Nov 01 20:30:44.460126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.457958    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:44.957944921 +0000 UTC m=+2.007054730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:44.490320 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.490302    4872 scope.go:122] "RemoveContainer" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0"
Nov 01 20:30:44.941372 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.941352    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:44.962179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962155    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:44.962296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962184    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:44.962296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962211    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:44.962296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962239    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.962296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962255    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:44.962296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962279    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962299    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962297    4872 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962314    4872 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962307    4872 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962366    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962348905 +0000 UTC m=+3.011458715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962378    4872 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962405    4872 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962382    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962375747 +0000 UTC m=+3.011485543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962357    4872 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962466    4872 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962469    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962453383 +0000 UTC m=+3.011563186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962497    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962475339 +0000 UTC m=+3.011585150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:44.962527 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.962528    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962613    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962597204 +0000 UTC m=+3.011707005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962623    4872 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962626    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962620059 +0000 UTC m=+3.011729855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962689    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962680887 +0000 UTC m=+3.011790683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962705    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962699803 +0000 UTC m=+3.011809599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : secret "metrics-daemon-secret" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962714    4872 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962723    4872 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.962749    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.962742903 +0000 UTC m=+3.011852699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.963033    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.963084    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.963170 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.963124    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:44.963524 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.963221    4872 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:44.963524 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:44.963268    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:45.963255402 +0000 UTC m=+3.012365202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:44.963524 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:44.963489    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10cbc89e-6b29-48e4-8218-03be6f3e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:45.132868 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.132845    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:30:45.135388 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.135368    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/1.log"
Nov 01 20:30:45.135475 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.135407    4872 generic.go:356] "Generic (PLEG): container finished" podID="b245f314-7357-4797-90d0-aa95f77b154a" containerID="3485c76089f519ec02faee5d86705aa737ea6297b642e4251805b829c9a356a8" exitCode=255
Nov 01 20:30:45.135513 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.135479    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" event={"ID":"b245f314-7357-4797-90d0-aa95f77b154a","Type":"ContainerDied","Data":"3485c76089f519ec02faee5d86705aa737ea6297b642e4251805b829c9a356a8"}
Nov 01 20:30:45.135580 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.135566    4872 scope.go:122] "RemoveContainer" containerID="a2bda48e267eb8b26f7474deabb22ac8c8abc814295da1ae44b1e7731fac70f0"
Nov 01 20:30:45.135882 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.135861    4872 scope.go:122] "RemoveContainer" containerID="3485c76089f519ec02faee5d86705aa737ea6297b642e4251805b829c9a356a8"
Nov 01 20:30:45.136068 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.136046    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9m5fw_openshift-console-operator(b245f314-7357-4797-90d0-aa95f77b154a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a"
Nov 01 20:30:45.970338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970308    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:45.970492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970346    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:45.970492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970374    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:45.970492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970410    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:45.970492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970439    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:45.970492 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970454    4872 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:45.970492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970485    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970496    4872 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970518    4872 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970503    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970464    4872 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970525    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.970506431 +0000 UTC m=+5.019616235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970556    4872 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970635    4872 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970638    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls podName:25f5b52b-e610-4b2f-9521-aec1e69f2170 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.970624911 +0000 UTC m=+5.019734709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls") pod "dns-default-glm2w" (UID: "25f5b52b-e610-4b2f-9521-aec1e69f2170") : secret "dns-default-metrics-tls" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970625    4872 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b48f6559f-rxmwf: secret "image-registry-tls" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970763    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.970742567 +0000 UTC m=+5.019852381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:45.970788 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970788    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.97077872 +0000 UTC m=+5.019888571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970808    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.97079888 +0000 UTC m=+5.019908679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970832    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs podName:f99c6ff8-8987-46e6-bfb3-8aea67369ea1 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.970823985 +0000 UTC m=+5.019933784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs") pod "network-metrics-daemon-cx7ds" (UID: "f99c6ff8-8987-46e6-bfb3-8aea67369ea1") : secret "metrics-daemon-secret" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.970850    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls podName:27ce2d60-a95b-4a6a-b793-1577fbb86703 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.970844346 +0000 UTC m=+5.019954142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls") pod "image-registry-6b48f6559f-rxmwf" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703") : secret "image-registry-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970900    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970943    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:45.970965    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971040    4872 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971051    4872 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971055    4872 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971069    4872 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54cb57cf74-86lck: secret "image-registry-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971095    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.971076497 +0000 UTC m=+5.020186300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971114    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.971104818 +0000 UTC m=+5.020214673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:45.971226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:45.971131    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls podName:8b26b214-b220-476f-af79-d819a9ba8c80 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.971121774 +0000 UTC m=+5.020231575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls") pod "image-registry-54cb57cf74-86lck" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80") : secret "image-registry-tls" not found
Nov 01 20:30:46.140507 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:46.140478    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:30:46.895116 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:46.895087    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:46.895286 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:46.895259    4872 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:30:47.362212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.362126    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-92gcr_6559ca8c-d167-4f33-95f9-1856858fc3d5/dns-node-resolver/0.log"
Nov 01 20:30:47.987463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987428    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987465    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987508    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987525    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987552    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987588    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987554    4872 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987614    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:47.987645 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987645    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987686    4872 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987697    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.987672713 +0000 UTC m=+9.036782525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987728    4872 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987733    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.987716398 +0000 UTC m=+9.036826199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987735    4872 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987764    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls podName:10cbc89e-6b29-48e4-8218-03be6f3e3992 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.987753202 +0000 UTC m=+9.036863004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntncn" (UID: "10cbc89e-6b29-48e4-8218-03be6f3e3992") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987786    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.987776916 +0000 UTC m=+9.036886727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987811    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.987795108 +0000 UTC m=+9.036904909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987862    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987893    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.987891    4872 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987944    4872 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:30:47.988126 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:47.987983    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:30:51.987973941 +0000 UTC m=+9.037083776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : secret "router-metrics-certs-default" not found
Nov 01 20:30:47.990117 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.990090    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c6ff8-8987-46e6-bfb3-8aea67369ea1-metrics-certs\") pod \"network-metrics-daemon-cx7ds\" (UID: \"f99c6ff8-8987-46e6-bfb3-8aea67369ea1\") " pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:47.990361 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.990335    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"image-registry-54cb57cf74-86lck\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") " pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:47.990521 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.990504    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25f5b52b-e610-4b2f-9521-aec1e69f2170-metrics-tls\") pod \"dns-default-glm2w\" (UID: \"25f5b52b-e610-4b2f-9521-aec1e69f2170\") " pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:47.990604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:47.990588    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"image-registry-6b48f6559f-rxmwf\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") " pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:48.025749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.025733    4872 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:48.026092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.026077    4872 scope.go:122] "RemoveContainer" containerID="3485c76089f519ec02faee5d86705aa737ea6297b642e4251805b829c9a356a8"
Nov 01 20:30:48.026257 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:48.026239    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9m5fw_openshift-console-operator(b245f314-7357-4797-90d0-aa95f77b154a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a"
Nov 01 20:30:48.091592 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.091576    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:48.097955 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.097931    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx7ds"
Nov 01 20:30:48.099649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.099621    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:48.100923 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.100903    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:48.561066 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.560908    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cx7ds"]
Nov 01 20:30:48.564135 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:48.564105    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c6ff8_8987_46e6_bfb3_8aea67369ea1.slice/crio-a04cbaf5f78630188de34c906bf8f9cabc582b4bf9a7913ad29f08f5723ec573 WatchSource:0}: Error finding container a04cbaf5f78630188de34c906bf8f9cabc582b4bf9a7913ad29f08f5723ec573: Status 404 returned error can't find the container with id a04cbaf5f78630188de34c906bf8f9cabc582b4bf9a7913ad29f08f5723ec573
Nov 01 20:30:48.564506 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.564486    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b48f6559f-rxmwf"]
Nov 01 20:30:48.569570 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.568553    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:30:48.569570 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.569089    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54cb57cf74-86lck"]
Nov 01 20:30:48.570561 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:48.570537    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ce2d60_a95b_4a6a_b793_1577fbb86703.slice/crio-240cb4c2a1b3b106232fea9f2f83cacb2d7a1dd9d72b45bc11749b02d610d659 WatchSource:0}: Error finding container 240cb4c2a1b3b106232fea9f2f83cacb2d7a1dd9d72b45bc11749b02d610d659: Status 404 returned error can't find the container with id 240cb4c2a1b3b106232fea9f2f83cacb2d7a1dd9d72b45bc11749b02d610d659
Nov 01 20:30:48.570696 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.570646    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w2766_ce97c7cb-9ccb-468e-af69-a8f8e922ec24/node-ca/0.log"
Nov 01 20:30:48.610206 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:48.610184    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glm2w"]
Nov 01 20:30:48.612574 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:48.612553    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b26b214_b220_476f_af79_d819a9ba8c80.slice/crio-6d8f97613eee67a268fa25b14df6134c4d9c9c3267a8662e48f00052158d6654 WatchSource:0}: Error finding container 6d8f97613eee67a268fa25b14df6134c4d9c9c3267a8662e48f00052158d6654: Status 404 returned error can't find the container with id 6d8f97613eee67a268fa25b14df6134c4d9c9c3267a8662e48f00052158d6654
Nov 01 20:30:48.615926 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:48.615904    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f5b52b_e610_4b2f_9521_aec1e69f2170.slice/crio-f4985d8a459be9a96322558e1de1393c6c6c2603e98ebc2ebc632b59a3c8267b WatchSource:0}: Error finding container f4985d8a459be9a96322558e1de1393c6c6c2603e98ebc2ebc632b59a3c8267b: Status 404 returned error can't find the container with id f4985d8a459be9a96322558e1de1393c6c6c2603e98ebc2ebc632b59a3c8267b
Nov 01 20:30:49.161091 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.161057    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf" event={"ID":"27ce2d60-a95b-4a6a-b793-1577fbb86703","Type":"ContainerStarted","Data":"26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77"}
Nov 01 20:30:49.161277 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.161110    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf" event={"ID":"27ce2d60-a95b-4a6a-b793-1577fbb86703","Type":"ContainerStarted","Data":"240cb4c2a1b3b106232fea9f2f83cacb2d7a1dd9d72b45bc11749b02d610d659"}
Nov 01 20:30:49.162220 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.162189    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glm2w" event={"ID":"25f5b52b-e610-4b2f-9521-aec1e69f2170","Type":"ContainerStarted","Data":"f4985d8a459be9a96322558e1de1393c6c6c2603e98ebc2ebc632b59a3c8267b"}
Nov 01 20:30:49.162628 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.162609    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwl2n_b60f90ff-66ee-410f-b953-aee5ed9b8d47/serve-healthcheck-canary/0.log"
Nov 01 20:30:49.163522 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.163499    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx7ds" event={"ID":"f99c6ff8-8987-46e6-bfb3-8aea67369ea1","Type":"ContainerStarted","Data":"a04cbaf5f78630188de34c906bf8f9cabc582b4bf9a7913ad29f08f5723ec573"}
Nov 01 20:30:49.164871 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.164851    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" event={"ID":"8b26b214-b220-476f-af79-d819a9ba8c80","Type":"ContainerStarted","Data":"2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df"}
Nov 01 20:30:49.164968 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.164885    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" event={"ID":"8b26b214-b220-476f-af79-d819a9ba8c80","Type":"ContainerStarted","Data":"6d8f97613eee67a268fa25b14df6134c4d9c9c3267a8662e48f00052158d6654"}
Nov 01 20:30:49.964580 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:49.964545    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn_5912e397-27c0-48e4-a25d-a902436d452d/kube-storage-version-migrator-operator/0.log"
Nov 01 20:30:50.177590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:50.177484    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx7ds" event={"ID":"f99c6ff8-8987-46e6-bfb3-8aea67369ea1","Type":"ContainerStarted","Data":"530e5a1b9815c4ec8a53e84f091fa959850f55e8430ce059d1debe203350994a"}
Nov 01 20:30:50.177590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:50.177510    4872 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:30:50.177815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:50.177791    4872 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:30:50.654623 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:50.654589    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:30:51.182267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.182169    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glm2w" event={"ID":"25f5b52b-e610-4b2f-9521-aec1e69f2170","Type":"ContainerStarted","Data":"e59e20213d445bdd14d0b77c8536eeab757f7482ed330f98740d8d3af771675b"}
Nov 01 20:30:51.182267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.182245    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glm2w" event={"ID":"25f5b52b-e610-4b2f-9521-aec1e69f2170","Type":"ContainerStarted","Data":"0e6e7c7e578a33936c9ae9bd8e706b65e4924dc6bf1e44b68a91bbd217d9a0e5"}
Nov 01 20:30:51.182748 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.182364    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-glm2w"
Nov 01 20:30:51.183988 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.183964    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx7ds" event={"ID":"f99c6ff8-8987-46e6-bfb3-8aea67369ea1","Type":"ContainerStarted","Data":"0111c65d645b81efcd67b8d0b1fa98968354641e21c953fb39c470bb1b77df26"}
Nov 01 20:30:51.715687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.715646    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:30:51.854295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.854265    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:51.854615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:51.854599    4872 scope.go:122] "RemoveContainer" containerID="3485c76089f519ec02faee5d86705aa737ea6297b642e4251805b829c9a356a8"
Nov 01 20:30:51.854809 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:51.854789    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9m5fw_openshift-console-operator(b245f314-7357-4797-90d0-aa95f77b154a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a"
Nov 01 20:30:52.022754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.022693    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:52.022754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.022734    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.022767    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.022802    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.022837    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.022887    4872 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.022909    4872 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.022940    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls podName:83552290-2377-4a32-be4c-e1f10c121b12 nodeName:}" failed. No retries permitted until 2026-11-01 20:31:00.022922944 +0000 UTC m=+17.072032764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7rw9k" (UID: "83552290-2377-4a32-be4c-e1f10c121b12") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.022954    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert podName:a0ba0752-5a6d-4490-a0bc-e3819c68a504 nodeName:}" failed. No retries permitted until 2026-11-01 20:31:00.022948454 +0000 UTC m=+17.072058250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sr6nr" (UID: "a0ba0752-5a6d-4490-a0bc-e3819c68a504") : secret "networking-console-plugin-cert" not found
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.023043    4872 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.023017    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:52.023103 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.023095    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls podName:c136bbc2-2ebc-40b7-8548-ca250b2dba7c nodeName:}" failed. No retries permitted until 2026-11-01 20:31:00.023079085 +0000 UTC m=+17.072188896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-flj4t" (UID: "c136bbc2-2ebc-40b7-8548-ca250b2dba7c") : secret "samples-operator-tls" not found
Nov 01 20:30:52.023430 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:30:52.023128    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle podName:b20d35a5-f97b-48a6-9dfb-73c6d656032c nodeName:}" failed. No retries permitted until 2026-11-01 20:31:00.023112318 +0000 UTC m=+17.072222120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle") pod "router-default-66ff46cf4c-rt6qk" (UID: "b20d35a5-f97b-48a6-9dfb-73c6d656032c") : configmap references non-existent config key: service-ca.crt
Nov 01 20:30:52.025006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.024985    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10cbc89e-6b29-48e4-8218-03be6f3e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntncn\" (UID: \"10cbc89e-6b29-48e4-8218-03be6f3e3992\") " pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:52.025082 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.025068    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20d35a5-f97b-48a6-9dfb-73c6d656032c-metrics-certs\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:30:52.300986 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.300927    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntncn"
Nov 01 20:30:52.688341 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:52.688316    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntncn"]
Nov 01 20:30:52.698494 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:30:52.698472    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cbc89e_6b29_48e4_8218_03be6f3e3992.slice/crio-c9c91c9e6e48cf52d64b907c1d011a915dc925968d129792f5a675e539e32975 WatchSource:0}: Error finding container c9c91c9e6e48cf52d64b907c1d011a915dc925968d129792f5a675e539e32975: Status 404 returned error can't find the container with id c9c91c9e6e48cf52d64b907c1d011a915dc925968d129792f5a675e539e32975
Nov 01 20:30:53.193938 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:53.193900    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntncn" event={"ID":"10cbc89e-6b29-48e4-8218-03be6f3e3992","Type":"ContainerStarted","Data":"a0a2ac09e1290c4955882f079bf264e4329f43554b9ef91450c300fc4b836872"}
Nov 01 20:30:53.194089 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:53.194005    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntncn" event={"ID":"10cbc89e-6b29-48e4-8218-03be6f3e3992","Type":"ContainerStarted","Data":"c9c91c9e6e48cf52d64b907c1d011a915dc925968d129792f5a675e539e32975"}
Nov 01 20:30:53.471941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:53.471870    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f7pr8"
Nov 01 20:30:53.615665 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:53.615637    4872 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-188.us-west-1.compute.internal" event="NodeReady"
Nov 01 20:30:54.198969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:54.198938    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntncn" event={"ID":"10cbc89e-6b29-48e4-8218-03be6f3e3992","Type":"ContainerStarted","Data":"72a8483ddce9d2944c7f753c94a5f2f0239ef49b49ea4c72621cd5344ba58fdd"}
Nov 01 20:30:54.998040 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:54.998011    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:30:56.071630 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.071605    4872 scope.go:122] "RemoveContainer" containerID="3485c76089f519ec02faee5d86705aa737ea6297b642e4251805b829c9a356a8"
Nov 01 20:30:56.211930 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.211907    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:30:56.212068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.211992    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" event={"ID":"b245f314-7357-4797-90d0-aa95f77b154a","Type":"ContainerStarted","Data":"0a0fd49d35a10336939a63f67b115468d88358890df707638b97e67f6a1b483a"}
Nov 01 20:30:56.212314 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.212289    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:30:56.212985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.212960    4872 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-9m5fw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Nov 01 20:30:56.213090 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.213006    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw" podUID="b245f314-7357-4797-90d0-aa95f77b154a" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Nov 01 20:30:56.214128 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.214104    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntncn" event={"ID":"10cbc89e-6b29-48e4-8218-03be6f3e3992","Type":"ContainerStarted","Data":"34edf27a79d5fa610b06a5c361042b4464461b2cbec0c4cba7bdb9d4d59c93ba"}
Nov 01 20:30:56.916197 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:56.916167    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9n"
Nov 01 20:30:57.221421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:30:57.221360    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9m5fw"
Nov 01 20:31:00.092411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.092376    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:31:00.092411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.092414    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:00.092946 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.092463    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:31:00.092946 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.092483    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:31:00.093246 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.093219    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b20d35a5-f97b-48a6-9dfb-73c6d656032c-service-ca-bundle\") pod \"router-default-66ff46cf4c-rt6qk\" (UID: \"b20d35a5-f97b-48a6-9dfb-73c6d656032c\") " pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:00.094749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.094718    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0ba0752-5a6d-4490-a0bc-e3819c68a504-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sr6nr\" (UID: \"a0ba0752-5a6d-4490-a0bc-e3819c68a504\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:31:00.094749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.094745    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c136bbc2-2ebc-40b7-8548-ca250b2dba7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-flj4t\" (UID: \"c136bbc2-2ebc-40b7-8548-ca250b2dba7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:31:00.094896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.094812    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/83552290-2377-4a32-be4c-e1f10c121b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7rw9k\" (UID: \"83552290-2377-4a32-be4c-e1f10c121b12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:31:00.388551 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.388488    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"
Nov 01 20:31:00.389585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.389568    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"
Nov 01 20:31:00.389691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.389680    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:00.391953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.391930    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"
Nov 01 20:31:00.767099 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.767069    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t"]
Nov 01 20:31:00.818192 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.818097    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k"]
Nov 01 20:31:00.818782 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.818765    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sr6nr"]
Nov 01 20:31:00.820764 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:31:00.820734    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83552290_2377_4a32_be4c_e1f10c121b12.slice/crio-9d5436f7340cfd9fd46609c2b812f0f46026848d2f268b78350df307a6f8799c WatchSource:0}: Error finding container 9d5436f7340cfd9fd46609c2b812f0f46026848d2f268b78350df307a6f8799c: Status 404 returned error can't find the container with id 9d5436f7340cfd9fd46609c2b812f0f46026848d2f268b78350df307a6f8799c
Nov 01 20:31:00.820978 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:31:00.820956    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ba0752_5a6d_4490_a0bc_e3819c68a504.slice/crio-8d440b6faa0d05da6f17bb5fbf503207b4e83cb16a3d4f384579e460ab84b273 WatchSource:0}: Error finding container 8d440b6faa0d05da6f17bb5fbf503207b4e83cb16a3d4f384579e460ab84b273: Status 404 returned error can't find the container with id 8d440b6faa0d05da6f17bb5fbf503207b4e83cb16a3d4f384579e460ab84b273
Nov 01 20:31:00.867349 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:00.867326    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66ff46cf4c-rt6qk"]
Nov 01 20:31:00.869917 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:31:00.869896    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20d35a5_f97b_48a6_9dfb_73c6d656032c.slice/crio-215f4535a5b3ae7eb7e458848fc108300cbf6141fcc683d2bb92c09911f27215 WatchSource:0}: Error finding container 215f4535a5b3ae7eb7e458848fc108300cbf6141fcc683d2bb92c09911f27215: Status 404 returned error can't find the container with id 215f4535a5b3ae7eb7e458848fc108300cbf6141fcc683d2bb92c09911f27215
Nov 01 20:31:01.190320 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.190298    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-glm2w"
Nov 01 20:31:01.231765 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.231732    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr" event={"ID":"a0ba0752-5a6d-4490-a0bc-e3819c68a504","Type":"ContainerStarted","Data":"8d440b6faa0d05da6f17bb5fbf503207b4e83cb16a3d4f384579e460ab84b273"}
Nov 01 20:31:01.232902 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.232881    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k" event={"ID":"83552290-2377-4a32-be4c-e1f10c121b12","Type":"ContainerStarted","Data":"9d5436f7340cfd9fd46609c2b812f0f46026848d2f268b78350df307a6f8799c"}
Nov 01 20:31:01.234036 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.234007    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t" event={"ID":"c136bbc2-2ebc-40b7-8548-ca250b2dba7c","Type":"ContainerStarted","Data":"c0444ff924d9cd03ad3162287d6431d2e438dd5388954e22ce36b4e8e29adeb0"}
Nov 01 20:31:01.235356 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.235334    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk" event={"ID":"b20d35a5-f97b-48a6-9dfb-73c6d656032c","Type":"ContainerStarted","Data":"dfb38be3a13e4f468fb53f19984fa1dba58a6645c79e1f497593282e4e7a1cf6"}
Nov 01 20:31:01.235445 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.235372    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk" event={"ID":"b20d35a5-f97b-48a6-9dfb-73c6d656032c","Type":"ContainerStarted","Data":"215f4535a5b3ae7eb7e458848fc108300cbf6141fcc683d2bb92c09911f27215"}
Nov 01 20:31:01.390581 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.390550    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:01.393177 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.393144    4872 patch_prober.go:29] interesting pod/router-default-66ff46cf4c-rt6qk container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Nov 01 20:31:01.393177 ip-10-0-131-188 kubenswrapper[4872]: [+]has-synced ok
Nov 01 20:31:01.393177 ip-10-0-131-188 kubenswrapper[4872]: [+]process-running ok
Nov 01 20:31:01.393177 ip-10-0-131-188 kubenswrapper[4872]: healthz check failed
Nov 01 20:31:01.393447 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.393196    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk" podUID="b20d35a5-f97b-48a6-9dfb-73c6d656032c" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 01 20:31:01.956278 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:01.956245    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b48f6559f-rxmwf"]
Nov 01 20:31:02.391835 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:02.391618    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:03.245264 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:03.245222    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sr6nr" event={"ID":"a0ba0752-5a6d-4490-a0bc-e3819c68a504","Type":"ContainerStarted","Data":"9cbf3e086a4ec1ed4d4b8d6c76c7ad17d66d8229578255e0daa57ec85df53032"}
Nov 01 20:31:03.246998 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:03.246968    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7rw9k" event={"ID":"83552290-2377-4a32-be4c-e1f10c121b12","Type":"ContainerStarted","Data":"49875a59609a18eed2df607a7daebedf792fb96eea1f0b44ee239d0a99211212"}
Nov 01 20:31:03.247255 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:03.247230    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:03.248600 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:03.248578    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66ff46cf4c-rt6qk"
Nov 01 20:31:03.997784 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:03.997752    4872 patch_prober.go:29] interesting pod/image-registry-54cb57cf74-86lck container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 20:31:03.998120 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:03.997812    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:31:04.252329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:04.252232    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t" event={"ID":"c136bbc2-2ebc-40b7-8548-ca250b2dba7c","Type":"ContainerStarted","Data":"28d06d2811449d9c2c2db50af5e08088dd14f50c0e67327045acc5a97b3dd556"}
Nov 01 20:31:04.252329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:04.252290    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-flj4t" event={"ID":"c136bbc2-2ebc-40b7-8548-ca250b2dba7c","Type":"ContainerStarted","Data":"df9e81e9b6a7aae10d150d00a32b8ce2f3af73cf9d6c9607f488f54425acf742"}
Nov 01 20:31:05.002348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:05.002313    4872 patch_prober.go:29] interesting pod/image-registry-54cb57cf74-86lck container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 20:31:05.002705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:05.002361    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:31:11.718806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:11.718773    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j4t6k"
Nov 01 20:31:11.961757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:11.961727    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:31:15.001984 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:15.001960    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:31:16.562132 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.562087    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7gmd9"]
Nov 01 20:31:16.592598 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.592569    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.647281 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.647248    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 20:31:16.647281 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.647264    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 20:31:16.697962 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.697928    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rcjlq\""
Nov 01 20:31:16.706323 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.706294    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 20:31:16.706684 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.706648    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 20:31:16.736835 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.736799    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-metrics-client-ca\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.736980 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.736841    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwr77\" (UniqueName: \"kubernetes.io/projected/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-kube-api-access-pwr77\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.736980 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.736874    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.736980 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.736942    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-textfile\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.737080 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.737009    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-tls\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.737112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.737078    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.737112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.737103    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-root\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.737172 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.737124    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-wtmp\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.737226 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.737207    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-sys\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.837786 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837706    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-textfile\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.837786 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837751    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-tls\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.837786 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837778    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837804    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-root\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837828    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-wtmp\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837859    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-sys\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837916    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-metrics-client-ca\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837928    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-root\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.837940    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr77\" (UniqueName: \"kubernetes.io/projected/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-kube-api-access-pwr77\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.838000    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.838020    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-sys\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.838040    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-textfile\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.838049    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-wtmp\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838594 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.838466    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.838594 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.838534    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-metrics-client-ca\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.840144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.840125    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.840337 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.840321    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-node-exporter-tls\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.899565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.899532    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwr77\" (UniqueName: \"kubernetes.io/projected/c2f1e2ba-c96e-4185-a335-c8f4e44ec422-kube-api-access-pwr77\") pod \"node-exporter-7gmd9\" (UID: \"c2f1e2ba-c96e-4185-a335-c8f4e44ec422\") " pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.942563 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:16.942526    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gmd9"
Nov 01 20:31:16.976563 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:31:16.976535    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f1e2ba_c96e_4185_a335_c8f4e44ec422.slice/crio-34ba4f4cb3c53fdde954a68f81979e714a8a8256d019e5d7c9a2fc0a2f0cf3a5 WatchSource:0}: Error finding container 34ba4f4cb3c53fdde954a68f81979e714a8a8256d019e5d7c9a2fc0a2f0cf3a5: Status 404 returned error can't find the container with id 34ba4f4cb3c53fdde954a68f81979e714a8a8256d019e5d7c9a2fc0a2f0cf3a5
Nov 01 20:31:17.300671 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:17.300617    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gmd9" event={"ID":"c2f1e2ba-c96e-4185-a335-c8f4e44ec422","Type":"ContainerStarted","Data":"34ba4f4cb3c53fdde954a68f81979e714a8a8256d019e5d7c9a2fc0a2f0cf3a5"}
Nov 01 20:31:19.313413 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:19.313374    4872 generic.go:356] "Generic (PLEG): container finished" podID="c2f1e2ba-c96e-4185-a335-c8f4e44ec422" containerID="e160ea2466ecd60f5ba7cb0c992d166b70d9772745481770b6a1300c3e0a3904" exitCode=0
Nov 01 20:31:19.313907 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:19.313421    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gmd9" event={"ID":"c2f1e2ba-c96e-4185-a335-c8f4e44ec422","Type":"ContainerDied","Data":"e160ea2466ecd60f5ba7cb0c992d166b70d9772745481770b6a1300c3e0a3904"}
Nov 01 20:31:20.319423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:20.319382    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gmd9" event={"ID":"c2f1e2ba-c96e-4185-a335-c8f4e44ec422","Type":"ContainerStarted","Data":"476d18a1a225a4a66bdda0b81237119bd98231be7e247d97f8634f77b9ba84e1"}
Nov 01 20:31:20.319890 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:20.319448    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gmd9" event={"ID":"c2f1e2ba-c96e-4185-a335-c8f4e44ec422","Type":"ContainerStarted","Data":"3e4798855f3e16fa9238ce4311384e24a8e3bae5582227c1ca57bc331a2a5478"}
Nov 01 20:31:20.443593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:20.443542    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7gmd9" podStartSLOduration=2.794828252 podStartE2EDuration="4.443527998s" totalImagesPullingTime="1.582433724s" totalInitContainerRuntime="66.266022ms" isStatefulPod=true podCreationTimestamp="2026-11-01 20:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:20.443073748 +0000 UTC m=+37.492183566" watchObservedRunningTime="2026-11-01 20:31:20.443527998 +0000 UTC m=+37.492637813"
Nov 01 20:31:23.366552 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:31:23.366511    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:31:24.687966 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:24.687936    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54cb57cf74-86lck"]
Nov 01 20:31:26.979644 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:26.979578    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf" podUID="27ce2d60-a95b-4a6a-b793-1577fbb86703" containerName="registry" containerID="cri-o://26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77" gracePeriod=30
Nov 01 20:31:27.240285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.240226    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:31:27.347265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.347230    4872 generic.go:356] "Generic (PLEG): container finished" podID="27ce2d60-a95b-4a6a-b793-1577fbb86703" containerID="26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77" exitCode=0
Nov 01 20:31:27.347436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.347272    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf"
Nov 01 20:31:27.347436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.347294    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf" event={"ID":"27ce2d60-a95b-4a6a-b793-1577fbb86703","Type":"ContainerDied","Data":"26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77"}
Nov 01 20:31:27.347436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.347338    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b48f6559f-rxmwf" event={"ID":"27ce2d60-a95b-4a6a-b793-1577fbb86703","Type":"ContainerDied","Data":"240cb4c2a1b3b106232fea9f2f83cacb2d7a1dd9d72b45bc11749b02d610d659"}
Nov 01 20:31:27.347436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.347358    4872 scope.go:122] "RemoveContainer" containerID="26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77"
Nov 01 20:31:27.355501 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.355479    4872 scope.go:122] "RemoveContainer" containerID="26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77"
Nov 01 20:31:27.355851 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:31:27.355830    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77\": container with ID starting with 26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77 not found: ID does not exist" containerID="26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77"
Nov 01 20:31:27.355914 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.355862    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77"} err="failed to get container status \"26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77\": rpc error: code = NotFound desc = could not find container \"26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77\": container with ID starting with 26cf295be55c51bc351b58d1d0113927462fcf5ad50d8268a2afa75b0f50fa77 not found: ID does not exist"
Nov 01 20:31:27.427023 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.426990    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427057    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427093    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427132    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427175    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427208    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427235    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427301    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") pod \"27ce2d60-a95b-4a6a-b793-1577fbb86703\" (UID: \"27ce2d60-a95b-4a6a-b793-1577fbb86703\") "
Nov 01 20:31:27.427638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427600    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:27.427764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.427731    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:27.430334 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.430300    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:27.435638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.430293    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:27.435638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.430773    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:27.435638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.430944    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "kube-api-access-dqqmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:27.435918 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.435858    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:27.438859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.438830    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted" pod "27ce2d60-a95b-4a6a-b793-1577fbb86703" (UID: "27ce2d60-a95b-4a6a-b793-1577fbb86703"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:27.528647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528561    4872 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-trusted-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528600    4872 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528613    4872 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ce2d60-a95b-4a6a-b793-1577fbb86703-ca-trust-extracted\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528622    4872 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ce2d60-a95b-4a6a-b793-1577fbb86703-registry-certificates\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528633    4872 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-installation-pull-secrets\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528642    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqqmj\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-kube-api-access-dqqmj\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528968 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528676    4872 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ce2d60-a95b-4a6a-b793-1577fbb86703-image-registry-private-configuration\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.528968 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.528702    4872 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ce2d60-a95b-4a6a-b793-1577fbb86703-bound-sa-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:27.767911 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.767876    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b48f6559f-rxmwf"]
Nov 01 20:31:27.770042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:27.770019    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b48f6559f-rxmwf"]
Nov 01 20:31:29.076597 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:29.076567    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ce2d60-a95b-4a6a-b793-1577fbb86703" path="/var/lib/kubelet/pods/27ce2d60-a95b-4a6a-b793-1577fbb86703/volumes"
Nov 01 20:31:49.707236 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.707173    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" containerName="registry" containerID="cri-o://2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df" gracePeriod=30
Nov 01 20:31:49.964073 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.964007    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:31:49.979193 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979160    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979355 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979239    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979355 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979297    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979355 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979325    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979375    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979403    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979445    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979514    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") pod \"8b26b214-b220-476f-af79-d819a9ba8c80\" (UID: \"8b26b214-b220-476f-af79-d819a9ba8c80\") "
Nov 01 20:31:49.979931 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.979838    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:49.981052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.980162    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:49.981973 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.981927    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:49.982066 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.981978    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:49.982110 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.982063    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:49.982110 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.982093    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "kube-api-access-5psvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:49.982175 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.982114    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:49.986518 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:49.986490    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted" pod "8b26b214-b220-476f-af79-d819a9ba8c80" (UID: "8b26b214-b220-476f-af79-d819a9ba8c80"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:50.080336 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080301    4872 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-registry-certificates\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080336 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080332    4872 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b26b214-b220-476f-af79-d819a9ba8c80-trusted-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080344    4872 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-image-registry-private-configuration\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080359    4872 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-registry-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080373    4872 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b26b214-b220-476f-af79-d819a9ba8c80-installation-pull-secrets\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080381    4872 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b26b214-b220-476f-af79-d819a9ba8c80-ca-trust-extracted\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080390    4872 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-bound-sa-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.080536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.080398    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psvg\" (UniqueName: \"kubernetes.io/projected/8b26b214-b220-476f-af79-d819a9ba8c80-kube-api-access-5psvg\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.441172 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.441135    4872 generic.go:356] "Generic (PLEG): container finished" podID="8b26b214-b220-476f-af79-d819a9ba8c80" containerID="2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df" exitCode=0
Nov 01 20:31:50.441371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.441204    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cb57cf74-86lck"
Nov 01 20:31:50.441371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.441220    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" event={"ID":"8b26b214-b220-476f-af79-d819a9ba8c80","Type":"ContainerDied","Data":"2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df"}
Nov 01 20:31:50.441371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.441276    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54cb57cf74-86lck" event={"ID":"8b26b214-b220-476f-af79-d819a9ba8c80","Type":"ContainerDied","Data":"6d8f97613eee67a268fa25b14df6134c4d9c9c3267a8662e48f00052158d6654"}
Nov 01 20:31:50.441371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.441297    4872 scope.go:122] "RemoveContainer" containerID="2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df"
Nov 01 20:31:50.449746 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.449729    4872 scope.go:122] "RemoveContainer" containerID="2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df"
Nov 01 20:31:50.450042 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:31:50.450025    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df\": container with ID starting with 2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df not found: ID does not exist" containerID="2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df"
Nov 01 20:31:50.450081 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.450051    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df"} err="failed to get container status \"2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df\": rpc error: code = NotFound desc = could not find container \"2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df\": container with ID starting with 2ac74ba984ee34722c2613b30f054c451126c8c4acd2a3092f794809034088df not found: ID does not exist"
Nov 01 20:31:50.606748 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.606714    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54cb57cf74-86lck"]
Nov 01 20:31:50.610218 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:50.610190    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54cb57cf74-86lck"]
Nov 01 20:31:51.076309 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:31:51.076277    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" path="/var/lib/kubelet/pods/8b26b214-b220-476f-af79-d819a9ba8c80/volumes"
Nov 01 20:31:53.364151 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:31:53.364114    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:32:03.492267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:03.492233    4872 generic.go:356] "Generic (PLEG): container finished" podID="8b5d9c2b-5435-40c7-bafb-a6b1b741e37b" containerID="a95c4325610e143250e71b12ce5457f36e709afdf05238d7477dacd7481630e2" exitCode=0
Nov 01 20:32:03.492716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:03.492305    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9rgh" event={"ID":"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b","Type":"ContainerDied","Data":"a95c4325610e143250e71b12ce5457f36e709afdf05238d7477dacd7481630e2"}
Nov 01 20:32:03.492716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:03.492624    4872 scope.go:122] "RemoveContainer" containerID="a95c4325610e143250e71b12ce5457f36e709afdf05238d7477dacd7481630e2"
Nov 01 20:32:04.498403 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:04.498365    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9rgh" event={"ID":"8b5d9c2b-5435-40c7-bafb-a6b1b741e37b","Type":"ContainerStarted","Data":"b01093eb08e45a1adf16ad778f7badc9e467cb3885c3156db7e7b90374871655"}
Nov 01 20:32:05.503305 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:05.503271    4872 generic.go:356] "Generic (PLEG): container finished" podID="5912e397-27c0-48e4-a25d-a902436d452d" containerID="5e253f536b9757f87e238576941e5981d01f1d2d98efaa61f0617856f6c2b251" exitCode=0
Nov 01 20:32:05.503715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:05.503352    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn" event={"ID":"5912e397-27c0-48e4-a25d-a902436d452d","Type":"ContainerDied","Data":"5e253f536b9757f87e238576941e5981d01f1d2d98efaa61f0617856f6c2b251"}
Nov 01 20:32:05.503715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:05.503711    4872 scope.go:122] "RemoveContainer" containerID="5e253f536b9757f87e238576941e5981d01f1d2d98efaa61f0617856f6c2b251"
Nov 01 20:32:06.509485 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:06.509447    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv2xn" event={"ID":"5912e397-27c0-48e4-a25d-a902436d452d","Type":"ContainerStarted","Data":"a6dc2403c5b7bde73fa8c03a6bcda7258144bbeee9c29f2e98b67b5973092603"}
Nov 01 20:32:14.539218 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:14.539181    4872 generic.go:356] "Generic (PLEG): container finished" podID="9cc3e550-0cb5-4301-82bc-17503dc53deb" containerID="513369be3d6e94f3e79e7bcae3e3481a1db33a8ad3a2219ea661eab71b9e09a5" exitCode=0
Nov 01 20:32:14.539602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:14.539243    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k" event={"ID":"9cc3e550-0cb5-4301-82bc-17503dc53deb","Type":"ContainerDied","Data":"513369be3d6e94f3e79e7bcae3e3481a1db33a8ad3a2219ea661eab71b9e09a5"}
Nov 01 20:32:14.539602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:14.539571    4872 scope.go:122] "RemoveContainer" containerID="513369be3d6e94f3e79e7bcae3e3481a1db33a8ad3a2219ea661eab71b9e09a5"
Nov 01 20:32:15.546515 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:32:15.546479    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5476k" event={"ID":"9cc3e550-0cb5-4301-82bc-17503dc53deb","Type":"ContainerStarted","Data":"5a70fb71eeb4dc992bafd9fe766cddfe0c0fe96cca9681b4293eff5f23678641"}
Nov 01 20:32:23.363947 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:32:23.363908    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:32:53.366118 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:32:53.366082    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:33:23.363482 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:33:23.363448    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:33:53.363591 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:33:53.363552    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:34:09.358983 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.358948    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"]
Nov 01 20:34:09.359414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.359240    4872 cpu_manager.go:414] "containerMap: removing container" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" containerName="registry"
Nov 01 20:34:09.359414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.359251    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" containerName="registry"
Nov 01 20:34:09.359414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.359266    4872 cpu_manager.go:414] "containerMap: removing container" podUID="27ce2d60-a95b-4a6a-b793-1577fbb86703" containerName="registry"
Nov 01 20:34:09.359414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.359271    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ce2d60-a95b-4a6a-b793-1577fbb86703" containerName="registry"
Nov 01 20:34:09.359414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.359331    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b26b214-b220-476f-af79-d819a9ba8c80" containerName="registry"
Nov 01 20:34:09.359414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.359341    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ce2d60-a95b-4a6a-b793-1577fbb86703" containerName="registry"
Nov 01 20:34:09.362181 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.362165    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.418024 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.417999    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 20:34:09.418024 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.418015    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 20:34:09.418024 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.418008    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 20:34:09.418248 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.418023    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 20:34:09.418248 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.418165    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z57nd\""
Nov 01 20:34:09.468261 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468230    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.468383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468273    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.468383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468354    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.468468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468389    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.468518 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468496    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzvj\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.468575 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468563    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.468624 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.468595    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.469925 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.469908    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"]
Nov 01 20:34:09.569805 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.569771    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzvj\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.569805 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.569811    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.569961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.569867    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.569961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.569906    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.569961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.569944    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.570064 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.569981    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.570064 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.570011    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.570668 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.570626    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.572324 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.572297    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.572416 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.572326    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.572416 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.572389    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.572487 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.572437    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.626741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.626690    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.626910 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.626888    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzvj\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj\") pod \"istiod-openshift-gateway-7876bfcb74-q97t2\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:09.672343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:09.672324    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:10.093714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:10.093686    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"]
Nov 01 20:34:10.134441 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:34:10.134415    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df354db_663a_42e7_a264_e0947f896323.slice/crio-3c5a080a68a9a8bd89aa67a242c949e6da029b2729c669141d10e3a9ecf58b04 WatchSource:0}: Error finding container 3c5a080a68a9a8bd89aa67a242c949e6da029b2729c669141d10e3a9ecf58b04: Status 404 returned error can't find the container with id 3c5a080a68a9a8bd89aa67a242c949e6da029b2729c669141d10e3a9ecf58b04
Nov 01 20:34:10.946892 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:10.946859    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" event={"ID":"8df354db-663a-42e7-a264-e0947f896323","Type":"ContainerStarted","Data":"3c5a080a68a9a8bd89aa67a242c949e6da029b2729c669141d10e3a9ecf58b04"}
Nov 01 20:34:14.194498 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:14.194470    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:34:14.194774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:14.194534    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:34:14.964020 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:14.963984    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" event={"ID":"8df354db-663a-42e7-a264-e0947f896323","Type":"ContainerStarted","Data":"b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4"}
Nov 01 20:34:14.964174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:14.964111    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:15.135201 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:15.135158    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" podStartSLOduration=2.077529433 podStartE2EDuration="6.135148036s" totalImagesPullingTime="4.057618603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:15.134879306 +0000 UTC m=+212.183989123" watchObservedRunningTime="2026-11-01 20:34:15.135148036 +0000 UTC m=+212.184257851"
Nov 01 20:34:15.969061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:15.969031    4872 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-q97t2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:34:15.969423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:15.969086    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" podUID="8df354db-663a-42e7-a264-e0947f896323" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:34:18.968769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:34:18.968739    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:34:23.362755 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:34:23.362713    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:34:53.362988 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:34:53.362952    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:35:00.154133 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.154102    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"]
Nov 01 20:35:00.157414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.157397    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.210139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.210103    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 20:35:00.210248 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.210129    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 20:35:00.210248 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.210167    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 20:35:00.210326 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.210285    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 20:35:00.210326 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.210311    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46fcz\""
Nov 01 20:35:00.210387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.210376    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 20:35:00.302184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.302158    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"]
Nov 01 20:35:00.312312 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.312290    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.312423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.312317    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6a138db-4d55-4d7c-a257-32dda7d6f963-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.312423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.312334    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.312423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.312353    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6a138db-4d55-4d7c-a257-32dda7d6f963-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.312545 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.312471    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whc8\" (UniqueName: \"kubernetes.io/projected/a6a138db-4d55-4d7c-a257-32dda7d6f963-kube-api-access-5whc8\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.312585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.312566    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413349    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413381    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413401    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6a138db-4d55-4d7c-a257-32dda7d6f963-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413417    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413436    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6a138db-4d55-4d7c-a257-32dda7d6f963-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413490    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whc8\" (UniqueName: \"kubernetes.io/projected/a6a138db-4d55-4d7c-a257-32dda7d6f963-kube-api-access-5whc8\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.413915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.413890    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6a138db-4d55-4d7c-a257-32dda7d6f963-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.415591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.415566    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6a138db-4d55-4d7c-a257-32dda7d6f963-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.415856 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.415837    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.415937 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.415895    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.415937 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.415898    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6a138db-4d55-4d7c-a257-32dda7d6f963-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.476732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.476710    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whc8\" (UniqueName: \"kubernetes.io/projected/a6a138db-4d55-4d7c-a257-32dda7d6f963-kube-api-access-5whc8\") pod \"opendatahub-operator-controller-manager-bd887889c-rcwr8\" (UID: \"a6a138db-4d55-4d7c-a257-32dda7d6f963\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:00.771975 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:00.771929    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:01.278905 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:01.278754    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"]
Nov 01 20:35:01.281567 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:35:01.281540    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a138db_4d55_4d7c_a257_32dda7d6f963.slice/crio-2648886ae8779baea66c2230233bf246138a6eb4ba2008db29400e5a0675f672 WatchSource:0}: Error finding container 2648886ae8779baea66c2230233bf246138a6eb4ba2008db29400e5a0675f672: Status 404 returned error can't find the container with id 2648886ae8779baea66c2230233bf246138a6eb4ba2008db29400e5a0675f672
Nov 01 20:35:02.115751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:02.115717    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8" event={"ID":"a6a138db-4d55-4d7c-a257-32dda7d6f963","Type":"ContainerStarted","Data":"2648886ae8779baea66c2230233bf246138a6eb4ba2008db29400e5a0675f672"}
Nov 01 20:35:07.135750 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:07.135680    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8" event={"ID":"a6a138db-4d55-4d7c-a257-32dda7d6f963","Type":"ContainerStarted","Data":"2caef3ccee0fb8e41044114ae966864c158bd7c1701c8a8eb91a96f20bc9721f"}
Nov 01 20:35:08.140616 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:08.140583    4872 generic.go:356] "Generic (PLEG): container finished" podID="a6a138db-4d55-4d7c-a257-32dda7d6f963" containerID="2caef3ccee0fb8e41044114ae966864c158bd7c1701c8a8eb91a96f20bc9721f" exitCode=0
Nov 01 20:35:08.140978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:08.140638    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8" event={"ID":"a6a138db-4d55-4d7c-a257-32dda7d6f963","Type":"ContainerDied","Data":"2caef3ccee0fb8e41044114ae966864c158bd7c1701c8a8eb91a96f20bc9721f"}
Nov 01 20:35:09.146172 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:09.146143    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8" event={"ID":"a6a138db-4d55-4d7c-a257-32dda7d6f963","Type":"ContainerStarted","Data":"56f9b39da9ddf0ec5f7bd4f1eb05d975c0d78be90d30d39122c4293fad5e733c"}
Nov 01 20:35:09.146566 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:09.146279    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:09.384423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:09.384347    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8" podStartSLOduration=3.542349115 podStartE2EDuration="10.384335163s" totalImagesPullingTime="6.428071823s" totalInitContainerRuntime="413.914225ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:09.333500565 +0000 UTC m=+266.382610383" watchObservedRunningTime="2026-11-01 20:35:09.384335163 +0000 UTC m=+266.433444980"
Nov 01 20:35:20.157447 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:20.157413    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-rcwr8"
Nov 01 20:35:23.364916 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:35:23.364869    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:35:32.316188 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.316152    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"]
Nov 01 20:35:32.319274 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.319256    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.422832 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.422795    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"]
Nov 01 20:35:32.440912 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.440879    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgtf\" (UniqueName: \"kubernetes.io/projected/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-kube-api-access-xtgtf\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.440912 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.440915    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-tls-certs\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.441092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.440950    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-tmp\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.475142 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.475112    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 20:35:32.475262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.475158    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hrkrz\""
Nov 01 20:35:32.484272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.484253    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 20:35:32.541792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.541763    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-tmp\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.541952 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.541817    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgtf\" (UniqueName: \"kubernetes.io/projected/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-kube-api-access-xtgtf\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.541952 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.541839    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-tls-certs\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.544061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.544038    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-tmp\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.544153 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.544108    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-tls-certs\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.598938 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.598876    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgtf\" (UniqueName: \"kubernetes.io/projected/aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc-kube-api-access-xtgtf\") pod \"kube-auth-proxy-85b9bf4598-hm78q\" (UID: \"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.630962 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.630933    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"
Nov 01 20:35:32.997983 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:32.997948    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q"]
Nov 01 20:35:33.001008 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:35:33.000985    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4e14e6_d5df_4f4a_8cd0_f876eb22cbbc.slice/crio-3c443c5cded5ab5970357f85421de854603900f236bfbdafd8340d41d06a03d6 WatchSource:0}: Error finding container 3c443c5cded5ab5970357f85421de854603900f236bfbdafd8340d41d06a03d6: Status 404 returned error can't find the container with id 3c443c5cded5ab5970357f85421de854603900f236bfbdafd8340d41d06a03d6
Nov 01 20:35:33.236170 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:33.236134    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q" event={"ID":"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc","Type":"ContainerStarted","Data":"3c443c5cded5ab5970357f85421de854603900f236bfbdafd8340d41d06a03d6"}
Nov 01 20:35:37.254852 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:37.254810    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q" event={"ID":"aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc","Type":"ContainerStarted","Data":"9f4b4f1e5a1e9efd2999bcd7af2ace84cd1619687b9eb71b73db8f61e294679d"}
Nov 01 20:35:37.370164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:37.370102    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-hm78q" podStartSLOduration=2.028516061 podStartE2EDuration="5.370088632s" totalImagesPullingTime="3.341572571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:37.36957653 +0000 UTC m=+294.418686348" watchObservedRunningTime="2026-11-01 20:35:37.370088632 +0000 UTC m=+294.419198449"
Nov 01 20:35:43.099060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:43.099027    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:35:43.099060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:43.099054    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:35:43.104180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:43.104159    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:35:43.104180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:43.104176    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:35:53.368451 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:35:53.368416    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:35:59.068759 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.068684    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"]
Nov 01 20:35:59.071879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.071856    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.140985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.140963    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f8tz\" (UniqueName: \"kubernetes.io/projected/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-kube-api-access-7f8tz\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.141085 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.141004    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-cert\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.141085 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.141029    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-metrics-certs\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.141164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.141111    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-manager-config\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.177936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.177919    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 20:35:59.178329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.178311    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 20:35:59.178409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.178317    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 20:35:59.178409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.178333    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 20:35:59.178409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.178333    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2njj2\""
Nov 01 20:35:59.178409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.178379    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:35:59.179007 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.178992    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"]
Nov 01 20:35:59.242128 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.242108    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f8tz\" (UniqueName: \"kubernetes.io/projected/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-kube-api-access-7f8tz\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.242216 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.242147    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-cert\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.242216 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.242184    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-metrics-certs\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.242216 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.242212    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-manager-config\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.242845 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.242824    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-manager-config\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.244444 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.244416    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-cert\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.244605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.244586    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-metrics-certs\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.300575 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.300554    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f8tz\" (UniqueName: \"kubernetes.io/projected/ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514-kube-api-access-7f8tz\") pod \"jobset-controller-manager-745695b8b-wpnlp\" (UID: \"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514\") " pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.383178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.383127    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:35:59.752147 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.752112    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"]
Nov 01 20:35:59.755120 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:35:59.755091    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc5ebd2_e814_4e71_bb2f_2d02f3b1a514.slice/crio-2bad9a4250c2a77b7be79dc1c6baf25777ed628fe9f2861ed46ad0bdd4cc63d1 WatchSource:0}: Error finding container 2bad9a4250c2a77b7be79dc1c6baf25777ed628fe9f2861ed46ad0bdd4cc63d1: Status 404 returned error can't find the container with id 2bad9a4250c2a77b7be79dc1c6baf25777ed628fe9f2861ed46ad0bdd4cc63d1
Nov 01 20:35:59.756951 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:35:59.756934    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:36:00.341908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:00.341874    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp" event={"ID":"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514","Type":"ContainerStarted","Data":"2bad9a4250c2a77b7be79dc1c6baf25777ed628fe9f2861ed46ad0bdd4cc63d1"}
Nov 01 20:36:05.362764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:05.362722    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp" event={"ID":"ccc5ebd2-e814-4e71-bb2f-2d02f3b1a514","Type":"ContainerStarted","Data":"7242526809b90de77a6c7e220bce7563f5159033acd4964c72de88696fdc8b76"}
Nov 01 20:36:05.363188 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:05.362848    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:36:05.477955 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:05.477904    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp" podStartSLOduration=1.871968208 podStartE2EDuration="6.477890258s" totalImagesPullingTime="4.60592205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:05.477105283 +0000 UTC m=+322.526215102" watchObservedRunningTime="2026-11-01 20:36:05.477890258 +0000 UTC m=+322.527000075"
Nov 01 20:36:16.372169 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:16.372139    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-745695b8b-wpnlp"
Nov 01 20:36:23.362533 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:36:23.362501    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:36:50.048448 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.048413    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-plqqn"]
Nov 01 20:36:50.054908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.054890    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.110558 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.110531    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-plqqn"]
Nov 01 20:36:50.131684 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.131644    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e597514c-f313-41a8-975e-06872e130081-metrics-certs\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.131784 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.131691    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zzz\" (UniqueName: \"kubernetes.io/projected/e597514c-f313-41a8-975e-06872e130081-kube-api-access-l8zzz\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.131784 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.131765    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e597514c-f313-41a8-975e-06872e130081-cert\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.159826 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.159805    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 20:36:50.159908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.159895    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 20:36:50.159977 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.159965    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 20:36:50.160028 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.160017    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9x47z\""
Nov 01 20:36:50.160060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.160042    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 20:36:50.232873 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.232847    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e597514c-f313-41a8-975e-06872e130081-cert\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.232961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.232894    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e597514c-f313-41a8-975e-06872e130081-metrics-certs\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.232961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.232909    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zzz\" (UniqueName: \"kubernetes.io/projected/e597514c-f313-41a8-975e-06872e130081-kube-api-access-l8zzz\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.235280 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.235261    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e597514c-f313-41a8-975e-06872e130081-metrics-certs\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.235366 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.235302    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e597514c-f313-41a8-975e-06872e130081-cert\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.290034 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.290011    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zzz\" (UniqueName: \"kubernetes.io/projected/e597514c-f313-41a8-975e-06872e130081-kube-api-access-l8zzz\") pod \"odh-model-controller-769875db56-plqqn\" (UID: \"e597514c-f313-41a8-975e-06872e130081\") " pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.366698 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.366639    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:50.736470 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:50.736438    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-plqqn"]
Nov 01 20:36:50.738981 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:36:50.738951    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode597514c_f313_41a8_975e_06872e130081.slice/crio-f661c8bf4863db75c74ec3901f591ecb3f71f62340f563151361ef823fde9c96 WatchSource:0}: Error finding container f661c8bf4863db75c74ec3901f591ecb3f71f62340f563151361ef823fde9c96: Status 404 returned error can't find the container with id f661c8bf4863db75c74ec3901f591ecb3f71f62340f563151361ef823fde9c96
Nov 01 20:36:51.548668 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:51.548631    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-plqqn" event={"ID":"e597514c-f313-41a8-975e-06872e130081","Type":"ContainerStarted","Data":"f661c8bf4863db75c74ec3901f591ecb3f71f62340f563151361ef823fde9c96"}
Nov 01 20:36:53.101792 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.101759    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"]
Nov 01 20:36:53.102224 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.101948    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" podUID="8df354db-663a-42e7-a264-e0947f896323" containerName="discovery" containerID="cri-o://b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4" gracePeriod=30
Nov 01 20:36:53.342903 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.342877    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:36:53.364699 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:36:53.364625    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:36:53.457133 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457106    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457278 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457164    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457278 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457182    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457285    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457339    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457372    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457550 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457458    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token\") pod \"8df354db-663a-42e7-a264-e0947f896323\" (UID: \"8df354db-663a-42e7-a264-e0947f896323\") "
Nov 01 20:36:53.457597 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457546    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:36:53.457718 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.457700    4872 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8df354db-663a-42e7-a264-e0947f896323-istio-csr-ca-configmap\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.459512 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.459484    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:36:53.459628 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.459534    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:36:53.459628 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.459615    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:36:53.459798 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.459774    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "kube-api-access-xmzvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:53.459851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.459820    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:53.459907 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.459889    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token" pod "8df354db-663a-42e7-a264-e0947f896323" (UID: "8df354db-663a-42e7-a264-e0947f896323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:53.557782 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.557753    4872 generic.go:356] "Generic (PLEG): container finished" podID="8df354db-663a-42e7-a264-e0947f896323" containerID="b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4" exitCode=0
Nov 01 20:36:53.557890 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.557809    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"
Nov 01 20:36:53.557890 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.557819    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" event={"ID":"8df354db-663a-42e7-a264-e0947f896323","Type":"ContainerDied","Data":"b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4"}
Nov 01 20:36:53.557890 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.557885    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2" event={"ID":"8df354db-663a-42e7-a264-e0947f896323","Type":"ContainerDied","Data":"3c5a080a68a9a8bd89aa67a242c949e6da029b2729c669141d10e3a9ecf58b04"}
Nov 01 20:36:53.557997 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.557908    4872 scope.go:122] "RemoveContainer" containerID="b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4"
Nov 01 20:36:53.558156 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.558113    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.558156 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.558140    4872 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-cacerts\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.558156 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.558154    4872 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-csr-dns-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.558309 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.558168    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmzvj\" (UniqueName: \"kubernetes.io/projected/8df354db-663a-42e7-a264-e0947f896323-kube-api-access-xmzvj\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.558309 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.558177    4872 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df354db-663a-42e7-a264-e0947f896323-istio-kubeconfig\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.558309 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.558185    4872 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df354db-663a-42e7-a264-e0947f896323-local-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:53.566208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.566194    4872 scope.go:122] "RemoveContainer" containerID="b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4"
Nov 01 20:36:53.566471 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:36:53.566456    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4\": container with ID starting with b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4 not found: ID does not exist" containerID="b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4"
Nov 01 20:36:53.566516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.566479    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4"} err="failed to get container status \"b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4\": rpc error: code = NotFound desc = could not find container \"b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4\": container with ID starting with b4e032f4e843a6acb935b7183f54b56aea96cf96b83318b5147afc5df55023b4 not found: ID does not exist"
Nov 01 20:36:53.723313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.723291    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"]
Nov 01 20:36:53.726884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:53.726864    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-q97t2"]
Nov 01 20:36:55.076713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:55.076678    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df354db-663a-42e7-a264-e0947f896323" path="/var/lib/kubelet/pods/8df354db-663a-42e7-a264-e0947f896323/volumes"
Nov 01 20:36:55.568526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:55.568488    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-plqqn" event={"ID":"e597514c-f313-41a8-975e-06872e130081","Type":"ContainerStarted","Data":"407755cdbc248275db07fc4a6cc6dc9c1228ec39d56acd73b19696d7f4858e6c"}
Nov 01 20:36:55.568690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:55.568635    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:36:55.685714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:36:55.685643    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-plqqn" podStartSLOduration=1.457286842 podStartE2EDuration="5.685628487s" totalImagesPullingTime="4.228341645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:55.684457339 +0000 UTC m=+372.733567157" watchObservedRunningTime="2026-11-01 20:36:55.685628487 +0000 UTC m=+372.734738343"
Nov 01 20:37:06.574677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:06.574627    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-plqqn"
Nov 01 20:37:07.244088 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.244054    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-k76ht"]
Nov 01 20:37:07.244363 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.244351    4872 cpu_manager.go:414] "containerMap: removing container" podUID="8df354db-663a-42e7-a264-e0947f896323" containerName="discovery"
Nov 01 20:37:07.244416 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.244364    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df354db-663a-42e7-a264-e0947f896323" containerName="discovery"
Nov 01 20:37:07.244451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.244426    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df354db-663a-42e7-a264-e0947f896323" containerName="discovery"
Nov 01 20:37:07.247169 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.247153    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.300311 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.300287    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nm6vj\""
Nov 01 20:37:07.300435 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.300397    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 20:37:07.351125 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.351101    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 20:37:07.351251 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.351234    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 20:37:07.352441 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.352423    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-k76ht"]
Nov 01 20:37:07.358598 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.358581    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.358667 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.358623    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvgk\" (UniqueName: \"kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.358667 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.358646    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.459907 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.459886    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.460005 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.459930    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvgk\" (UniqueName: \"kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.460005 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.459952    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.462217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.462195    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.462310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.462249    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.517727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.517669    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvgk\" (UniqueName: \"kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk\") pod \"maas-controller-579cf7c6d-k76ht\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") " pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.558733 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.558714    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:07.640963 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.640931    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d79b4778f-ksf48"]
Nov 01 20:37:07.644840 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.644823    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.702250 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.702223    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4778f-ksf48"]
Nov 01 20:37:07.762412 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.762386    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fb7\" (UniqueName: \"kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.762519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.762432    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.762519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.762506    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.863692 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.863623    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fb7\" (UniqueName: \"kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.863788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.863773    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.863828 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.863803    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.866175 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.866151    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.866259 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.866175    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.920689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.920649    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fb7\" (UniqueName: \"kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7\") pod \"maas-controller-d79b4778f-ksf48\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") " pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:07.925590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.925571    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-k76ht"]
Nov 01 20:37:07.927873 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:37:07.927852    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8475681e_82cf_4e00_957b_7902dea113ca.slice/crio-26ccd61c0b8db0967a03287fc6eca5d2a218a1d49ca1d269c5aea7dcc233474a WatchSource:0}: Error finding container 26ccd61c0b8db0967a03287fc6eca5d2a218a1d49ca1d269c5aea7dcc233474a: Status 404 returned error can't find the container with id 26ccd61c0b8db0967a03287fc6eca5d2a218a1d49ca1d269c5aea7dcc233474a
Nov 01 20:37:07.956771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:07.956748    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:08.324050 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:08.323978    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4778f-ksf48"]
Nov 01 20:37:08.326526 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:37:08.326498    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054ff4af_b3f9_4652_b079_549bc614afa8.slice/crio-e14b24ca942825fb6e1c9119bf12246f267f0d13fee88b7e258e9d6d3004a32e WatchSource:0}: Error finding container e14b24ca942825fb6e1c9119bf12246f267f0d13fee88b7e258e9d6d3004a32e: Status 404 returned error can't find the container with id e14b24ca942825fb6e1c9119bf12246f267f0d13fee88b7e258e9d6d3004a32e
Nov 01 20:37:08.626458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:08.626400    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-ksf48" event={"ID":"054ff4af-b3f9-4652-b079-549bc614afa8","Type":"ContainerStarted","Data":"e14b24ca942825fb6e1c9119bf12246f267f0d13fee88b7e258e9d6d3004a32e"}
Nov 01 20:37:08.627619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:08.627597    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-k76ht" event={"ID":"8475681e-82cf-4e00-957b-7902dea113ca","Type":"ContainerStarted","Data":"26ccd61c0b8db0967a03287fc6eca5d2a218a1d49ca1d269c5aea7dcc233474a"}
Nov 01 20:37:13.652835 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:13.652791    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-k76ht" event={"ID":"8475681e-82cf-4e00-957b-7902dea113ca","Type":"ContainerStarted","Data":"88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67"}
Nov 01 20:37:13.653296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:13.652872    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:13.654445 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:13.654417    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-ksf48" event={"ID":"054ff4af-b3f9-4652-b079-549bc614afa8","Type":"ContainerStarted","Data":"f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41"}
Nov 01 20:37:13.654550 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:13.654525    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:13.769902 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:13.769858    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-k76ht" podStartSLOduration=1.66598215 podStartE2EDuration="6.769846052s" totalImagesPullingTime="5.103863902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:13.769224012 +0000 UTC m=+390.818333830" watchObservedRunningTime="2026-11-01 20:37:13.769846052 +0000 UTC m=+390.818955870"
Nov 01 20:37:13.886904 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:13.886862    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d79b4778f-ksf48" podStartSLOduration=2.190029979 podStartE2EDuration="6.886853034s" totalImagesPullingTime="4.696823055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:13.885158724 +0000 UTC m=+390.934268532" watchObservedRunningTime="2026-11-01 20:37:13.886853034 +0000 UTC m=+390.935962894"
Nov 01 20:37:16.687760 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.687725    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"]
Nov 01 20:37:16.691097 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.691078    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.748049 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.748016    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"]
Nov 01 20:37:16.796928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.796903    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 20:37:16.796928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.796889    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 20:37:16.797084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.796956    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-86d4w\""
Nov 01 20:37:16.797084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.796967    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 20:37:16.837980 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.837953    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.838100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.838001    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4641e48-f529-4403-b5ab-cb9c7537970e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.838100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.838017    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sln\" (UniqueName: \"kubernetes.io/projected/f4641e48-f529-4403-b5ab-cb9c7537970e-kube-api-access-78sln\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.838177 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.838115    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.940277 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.940216    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.940277 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.940262    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.940506 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.940303    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4641e48-f529-4403-b5ab-cb9c7537970e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.940506 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:16.940370    4872 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 20:37:16.940506 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:16.940397    4872 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 20:37:16.940506 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.940420    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sln\" (UniqueName: \"kubernetes.io/projected/f4641e48-f529-4403-b5ab-cb9c7537970e-kube-api-access-78sln\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:16.940506 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:16.940448    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-proxy-source podName:f4641e48-f529-4403-b5ab-cb9c7537970e nodeName:}" failed. No retries permitted until 2026-11-01 20:37:17.440430781 +0000 UTC m=+394.489540584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-tkz8s" (UID: "f4641e48-f529-4403-b5ab-cb9c7537970e") : configmap "usage-tenancy-proxy-source" not found
Nov 01 20:37:16.940506 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:16.940489    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4641e48-f529-4403-b5ab-cb9c7537970e-tls-cert podName:f4641e48-f529-4403-b5ab-cb9c7537970e nodeName:}" failed. No retries permitted until 2026-11-01 20:37:17.440461267 +0000 UTC m=+394.489571088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/f4641e48-f529-4403-b5ab-cb9c7537970e-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-tkz8s" (UID: "f4641e48-f529-4403-b5ab-cb9c7537970e") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 20:37:16.940970 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:16.940950    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.000042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.000019    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sln\" (UniqueName: \"kubernetes.io/projected/f4641e48-f529-4403-b5ab-cb9c7537970e-kube-api-access-78sln\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.445578 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.445549    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.445743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.445597    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4641e48-f529-4403-b5ab-cb9c7537970e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.446286 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.446262    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f4641e48-f529-4403-b5ab-cb9c7537970e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.447754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.447730    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4641e48-f529-4403-b5ab-cb9c7537970e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tkz8s\" (UID: \"f4641e48-f529-4403-b5ab-cb9c7537970e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.603093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.603065    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:17.977135 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:17.975976    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"]
Nov 01 20:37:18.677772 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:18.677728    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s" event={"ID":"f4641e48-f529-4403-b5ab-cb9c7537970e","Type":"ContainerStarted","Data":"8eb2be868de1f944667f2c64eb99e1f59929145e1fb3d393981e171877627d84"}
Nov 01 20:37:23.363015 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:23.362982    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:37:24.661106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:24.661076    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 20:37:24.662291 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:24.662247    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:24.833536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:24.833172    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-k76ht"]
Nov 01 20:37:24.833536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:24.833376    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-k76ht" podUID="8475681e-82cf-4e00-957b-7902dea113ca" containerName="manager" containerID="cri-o://88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67" gracePeriod=30
Nov 01 20:37:25.111891 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.111612    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:25.206427 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.206343    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs\") pod \"8475681e-82cf-4e00-957b-7902dea113ca\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") "
Nov 01 20:37:25.206427 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.206409    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert\") pod \"8475681e-82cf-4e00-957b-7902dea113ca\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") "
Nov 01 20:37:25.206602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.206543    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk\" (UniqueName: \"kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk\") pod \"8475681e-82cf-4e00-957b-7902dea113ca\" (UID: \"8475681e-82cf-4e00-957b-7902dea113ca\") "
Nov 01 20:37:25.210253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.210224    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs" pod "8475681e-82cf-4e00-957b-7902dea113ca" (UID: "8475681e-82cf-4e00-957b-7902dea113ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:37:25.210385 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.210348    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk" pod "8475681e-82cf-4e00-957b-7902dea113ca" (UID: "8475681e-82cf-4e00-957b-7902dea113ca"). InnerVolumeSpecName "kube-api-access-2qvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:37:25.211847 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.211812    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert" pod "8475681e-82cf-4e00-957b-7902dea113ca" (UID: "8475681e-82cf-4e00-957b-7902dea113ca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:37:25.307294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.307261    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qvgk\" (UniqueName: \"kubernetes.io/projected/8475681e-82cf-4e00-957b-7902dea113ca-kube-api-access-2qvgk\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:37:25.307456 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.307299    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:37:25.307456 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.307314    4872 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8475681e-82cf-4e00-957b-7902dea113ca-webhook-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:37:25.719068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.715065    4872 generic.go:356] "Generic (PLEG): container finished" podID="8475681e-82cf-4e00-957b-7902dea113ca" containerID="88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67" exitCode=0
Nov 01 20:37:25.719068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.715119    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-k76ht" event={"ID":"8475681e-82cf-4e00-957b-7902dea113ca","Type":"ContainerDied","Data":"88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67"}
Nov 01 20:37:25.719068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.715160    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-k76ht" event={"ID":"8475681e-82cf-4e00-957b-7902dea113ca","Type":"ContainerDied","Data":"26ccd61c0b8db0967a03287fc6eca5d2a218a1d49ca1d269c5aea7dcc233474a"}
Nov 01 20:37:25.719068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.715184    4872 scope.go:122] "RemoveContainer" containerID="88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67"
Nov 01 20:37:25.719068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.715299    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-k76ht"
Nov 01 20:37:25.729303 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.729283    4872 scope.go:122] "RemoveContainer" containerID="88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67"
Nov 01 20:37:25.729877 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:25.729846    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67\": container with ID starting with 88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67 not found: ID does not exist" containerID="88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67"
Nov 01 20:37:25.729972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.729891    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67"} err="failed to get container status \"88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67\": rpc error: code = NotFound desc = could not find container \"88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67\": container with ID starting with 88df5d40c72088c97dc7f92d39a375c93972c467fa5524a4626c7f522ee75a67 not found: ID does not exist"
Nov 01 20:37:25.889756 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.889716    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-k76ht"]
Nov 01 20:37:25.936917 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:25.936874    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-k76ht"]
Nov 01 20:37:27.078671 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:27.078624    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8475681e-82cf-4e00-957b-7902dea113ca" path="/var/lib/kubelet/pods/8475681e-82cf-4e00-957b-7902dea113ca/volumes"
Nov 01 20:37:33.752443 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:33.752411    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s" event={"ID":"f4641e48-f529-4403-b5ab-cb9c7537970e","Type":"ContainerStarted","Data":"38b9f5965ab23d8436e717e745a3f80c19f4a65d7e1b8f142197ae5d61583553"}
Nov 01 20:37:33.752867 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:33.752544    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:33.868124 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:33.868080    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s" podStartSLOduration=2.532231865 podStartE2EDuration="17.868067374s" totalImagesPullingTime="15.335835509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:33.867064592 +0000 UTC m=+410.916174410" watchObservedRunningTime="2026-11-01 20:37:33.868067374 +0000 UTC m=+410.917177233"
Nov 01 20:37:44.763203 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:44.763165    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tkz8s"
Nov 01 20:37:50.097955 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.097910    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"]
Nov 01 20:37:50.098449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.098279    4872 cpu_manager.go:414] "containerMap: removing container" podUID="8475681e-82cf-4e00-957b-7902dea113ca" containerName="manager"
Nov 01 20:37:50.098449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.098292    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8475681e-82cf-4e00-957b-7902dea113ca" containerName="manager"
Nov 01 20:37:50.098449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.098393    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="8475681e-82cf-4e00-957b-7902dea113ca" containerName="manager"
Nov 01 20:37:50.103484 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.103461    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.201182 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.201155    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d6j\" (UniqueName: \"kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.201303 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.201203    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.201303 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.201271    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.201394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.201321    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.209124 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.209092    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:37:50.209252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.209157    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:37:50.209312 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.209294    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cms9v\""
Nov 01 20:37:50.209379 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.209340    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 20:37:50.209502 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.209455    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 20:37:50.210722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.210704    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"]
Nov 01 20:37:50.301707 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.301686    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.301816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.301714    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.301816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.301744    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.301936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.301809    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d6j\" (UniqueName: \"kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.302417 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.302393    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.304138 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.304114    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.304212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.304187    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.359028 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.358974    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d6j\" (UniqueName: \"kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j\") pod \"maas-api-6d669c7fcd-9cn46\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.416809 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.416792    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:50.786397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.786367    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"]
Nov 01 20:37:50.788848 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:37:50.788818    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1016d1c_7da5_4d16_840c_f3101815bdea.slice/crio-b1c8e632a2e906d978a40fdb2105b6724c8aa6180342c50d7de10cf2d251031e WatchSource:0}: Error finding container b1c8e632a2e906d978a40fdb2105b6724c8aa6180342c50d7de10cf2d251031e: Status 404 returned error can't find the container with id b1c8e632a2e906d978a40fdb2105b6724c8aa6180342c50d7de10cf2d251031e
Nov 01 20:37:50.828222 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:50.828193    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46" event={"ID":"d1016d1c-7da5-4d16-840c-f3101815bdea","Type":"ContainerStarted","Data":"b1c8e632a2e906d978a40fdb2105b6724c8aa6180342c50d7de10cf2d251031e"}
Nov 01 20:37:53.363467 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:37:53.363425    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:37:55.850074 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:55.850043    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46" event={"ID":"d1016d1c-7da5-4d16-840c-f3101815bdea","Type":"ContainerStarted","Data":"d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7"}
Nov 01 20:37:55.850409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:55.850130    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:37:56.009226 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:37:56.009139    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46" podStartSLOduration=1.068800169 podStartE2EDuration="6.009127089s" totalImagesPullingTime="4.94032692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:56.008689081 +0000 UTC m=+433.057798901" watchObservedRunningTime="2026-11-01 20:37:56.009127089 +0000 UTC m=+433.058236906"
Nov 01 20:38:01.859006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:01.858975    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:38:02.816238 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:02.816205    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"]
Nov 01 20:38:03.058255 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.058225    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"]
Nov 01 20:38:03.058696 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.058337    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.098146 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.098089    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.098146 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.098119    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2tw\" (UniqueName: \"kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.098146 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.098140    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.098306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.098203    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.199173 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.199152    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.199295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.199179    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2tw\" (UniqueName: \"kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.199295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.199196    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.199295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.199213    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.199915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.199883    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.201596 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.201575    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.201596 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.201588    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.256033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.256013    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2tw\" (UniqueName: \"kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw\") pod \"maas-api-5f57b8b55f-bxpz2\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") " pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.388856 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.388795    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:03.772292 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.772264    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"]
Nov 01 20:38:03.774475 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:38:03.774450    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963e514d_5cb6_4a7c_b70f_2c8592186f04.slice/crio-a9816e4c3d146c2634c0269c0423c9b34160d32a5e03d79d95282fd0d82e16ea WatchSource:0}: Error finding container a9816e4c3d146c2634c0269c0423c9b34160d32a5e03d79d95282fd0d82e16ea: Status 404 returned error can't find the container with id a9816e4c3d146c2634c0269c0423c9b34160d32a5e03d79d95282fd0d82e16ea
Nov 01 20:38:03.883599 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:03.883575    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2" event={"ID":"963e514d-5cb6-4a7c-b70f-2c8592186f04","Type":"ContainerStarted","Data":"a9816e4c3d146c2634c0269c0423c9b34160d32a5e03d79d95282fd0d82e16ea"}
Nov 01 20:38:04.890217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:04.890188    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2" event={"ID":"963e514d-5cb6-4a7c-b70f-2c8592186f04","Type":"ContainerStarted","Data":"8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17"}
Nov 01 20:38:04.890559 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:04.890302    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:05.033633 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:05.033588    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2" podStartSLOduration=2.010097764 podStartE2EDuration="3.033575366s" totalImagesPullingTime="1.023477602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:05.031921185 +0000 UTC m=+442.081031004" watchObservedRunningTime="2026-11-01 20:38:05.033575366 +0000 UTC m=+442.082685184"
Nov 01 20:38:10.899052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:10.899022    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 20:38:11.067025 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.066992    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"]
Nov 01 20:38:11.067241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.067218    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46" podUID="d1016d1c-7da5-4d16-840c-f3101815bdea" containerName="maas-api" containerID="cri-o://d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7" gracePeriod=30
Nov 01 20:38:11.300883 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.300860    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:38:11.363782 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.363757    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs\") pod \"d1016d1c-7da5-4d16-840c-f3101815bdea\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") "
Nov 01 20:38:11.363877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.363824    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca\") pod \"d1016d1c-7da5-4d16-840c-f3101815bdea\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") "
Nov 01 20:38:11.363919 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.363881    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j\" (UniqueName: \"kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j\") pod \"d1016d1c-7da5-4d16-840c-f3101815bdea\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") "
Nov 01 20:38:11.363969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.363954    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls\") pod \"d1016d1c-7da5-4d16-840c-f3101815bdea\" (UID: \"d1016d1c-7da5-4d16-840c-f3101815bdea\") "
Nov 01 20:38:11.364227 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.364198    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca" pod "d1016d1c-7da5-4d16-840c-f3101815bdea" (UID: "d1016d1c-7da5-4d16-840c-f3101815bdea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:11.365936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.365913    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs" pod "d1016d1c-7da5-4d16-840c-f3101815bdea" (UID: "d1016d1c-7da5-4d16-840c-f3101815bdea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:11.366017 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.365943    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls" pod "d1016d1c-7da5-4d16-840c-f3101815bdea" (UID: "d1016d1c-7da5-4d16-840c-f3101815bdea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:11.366060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.366033    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j" pod "d1016d1c-7da5-4d16-840c-f3101815bdea" (UID: "d1016d1c-7da5-4d16-840c-f3101815bdea"). InnerVolumeSpecName "kube-api-access-49d6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:11.464881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.464824    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49d6j\" (UniqueName: \"kubernetes.io/projected/d1016d1c-7da5-4d16-840c-f3101815bdea-kube-api-access-49d6j\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:11.464881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.464849    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:11.464881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.464859    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1016d1c-7da5-4d16-840c-f3101815bdea-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:11.464881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.464868    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1016d1c-7da5-4d16-840c-f3101815bdea-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:11.917484 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.917446    4872 generic.go:356] "Generic (PLEG): container finished" podID="d1016d1c-7da5-4d16-840c-f3101815bdea" containerID="d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7" exitCode=0
Nov 01 20:38:11.917971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.917489    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"
Nov 01 20:38:11.917971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.917487    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46" event={"ID":"d1016d1c-7da5-4d16-840c-f3101815bdea","Type":"ContainerDied","Data":"d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7"}
Nov 01 20:38:11.917971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.917601    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46" event={"ID":"d1016d1c-7da5-4d16-840c-f3101815bdea","Type":"ContainerDied","Data":"b1c8e632a2e906d978a40fdb2105b6724c8aa6180342c50d7de10cf2d251031e"}
Nov 01 20:38:11.917971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.917628    4872 scope.go:122] "RemoveContainer" containerID="d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7"
Nov 01 20:38:11.926139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.926123    4872 scope.go:122] "RemoveContainer" containerID="d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7"
Nov 01 20:38:11.926385 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:38:11.926368    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7\": container with ID starting with d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7 not found: ID does not exist" containerID="d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7"
Nov 01 20:38:11.926437 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:11.926392    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7"} err="failed to get container status \"d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7\": rpc error: code = NotFound desc = could not find container \"d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7\": container with ID starting with d19de5e18efcccf6949966e33f590103dfc84e2f84545c4c08025590153a53a7 not found: ID does not exist"
Nov 01 20:38:12.081403 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:12.081381    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"]
Nov 01 20:38:12.132133 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:12.132109    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-9cn46"]
Nov 01 20:38:13.075725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:38:13.075689    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1016d1c-7da5-4d16-840c-f3101815bdea" path="/var/lib/kubelet/pods/d1016d1c-7da5-4d16-840c-f3101815bdea/volumes"
Nov 01 20:38:23.363456 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:38:23.363421    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:38:53.365216 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:38:53.365180    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:39:04.149519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.149448    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"]
Nov 01 20:39:04.149880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.149801    4872 cpu_manager.go:414] "containerMap: removing container" podUID="d1016d1c-7da5-4d16-840c-f3101815bdea" containerName="maas-api"
Nov 01 20:39:04.149880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.149814    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1016d1c-7da5-4d16-840c-f3101815bdea" containerName="maas-api"
Nov 01 20:39:04.149953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.149881    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1016d1c-7da5-4d16-840c-f3101815bdea" containerName="maas-api"
Nov 01 20:39:04.152731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.152711    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.241591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.241565    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.241754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.241603    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.241754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.241626    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.241866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.241756    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsqj\" (UniqueName: \"kubernetes.io/projected/d99c741f-b128-4269-81e1-af775684f2c0-kube-api-access-knsqj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.241866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.241827    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.241942 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.241885    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99c741f-b128-4269-81e1-af775684f2c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.256886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.256850    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p9lqs\""
Nov 01 20:39:04.257144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.257098    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 20:39:04.257144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.257100    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 20:39:04.257144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.257125    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 20:39:04.258359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.258343    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"]
Nov 01 20:39:04.343087 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343063    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343173 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343097    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343173 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343121    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343173 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343161    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsqj\" (UniqueName: \"kubernetes.io/projected/d99c741f-b128-4269-81e1-af775684f2c0-kube-api-access-knsqj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343183    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343202    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99c741f-b128-4269-81e1-af775684f2c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343505 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343480    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343593    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.343689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.343597    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.345251 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.345231    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99c741f-b128-4269-81e1-af775684f2c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.345408 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.345391    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99c741f-b128-4269-81e1-af775684f2c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.400253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.400202    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsqj\" (UniqueName: \"kubernetes.io/projected/d99c741f-b128-4269-81e1-af775684f2c0-kube-api-access-knsqj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv\" (UID: \"d99c741f-b128-4269-81e1-af775684f2c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.464039 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.464017    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:04.834839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:04.834766    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"]
Nov 01 20:39:04.837453 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:04.837429    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99c741f_b128_4269_81e1_af775684f2c0.slice/crio-afae3e4dd287a58318ad4cc0874e4d19b62da9dcd28a67a76772df5529b76978 WatchSource:0}: Error finding container afae3e4dd287a58318ad4cc0874e4d19b62da9dcd28a67a76772df5529b76978: Status 404 returned error can't find the container with id afae3e4dd287a58318ad4cc0874e4d19b62da9dcd28a67a76772df5529b76978
Nov 01 20:39:05.118641 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:05.118577    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv" event={"ID":"d99c741f-b128-4269-81e1-af775684f2c0","Type":"ContainerStarted","Data":"afae3e4dd287a58318ad4cc0874e4d19b62da9dcd28a67a76772df5529b76978"}
Nov 01 20:39:07.799803 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.799764    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"]
Nov 01 20:39:07.803325 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.803302    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:07.908024 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.907998    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 01 20:39:07.909368 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.909348    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"]
Nov 01 20:39:07.973592 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.973567    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fqqz\" (UniqueName: \"kubernetes.io/projected/7cfc3889-952c-448a-98ef-8098113fa769-kube-api-access-8fqqz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:07.973706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.973603    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:07.973706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.973624    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfc3889-952c-448a-98ef-8098113fa769-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:07.973778 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.973715    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:07.973811 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.973769    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:07.973811 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:07.973805    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075004    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075032    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075052    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075225 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075110    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fqqz\" (UniqueName: \"kubernetes.io/projected/7cfc3889-952c-448a-98ef-8098113fa769-kube-api-access-8fqqz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075225 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075150    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075225 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075172    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfc3889-952c-448a-98ef-8098113fa769-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075460 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075441    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075538 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075518    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.075595 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.075565    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.077217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.077188    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfc3889-952c-448a-98ef-8098113fa769-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.077613 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.077592    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfc3889-952c-448a-98ef-8098113fa769-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.132414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.132388    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv" event={"ID":"d99c741f-b128-4269-81e1-af775684f2c0","Type":"ContainerStarted","Data":"ff4d3324275756af29e6ef1ae7cf282cd046410eadf8892f47dac84806d3cc4e"}
Nov 01 20:39:08.134819 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.134798    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fqqz\" (UniqueName: \"kubernetes.io/projected/7cfc3889-952c-448a-98ef-8098113fa769-kube-api-access-8fqqz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-t4dq6\" (UID: \"7cfc3889-952c-448a-98ef-8098113fa769\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.247740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.247692    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv" podStartSLOduration=1.918272097 podStartE2EDuration="4.247676868s" totalImagesPullingTime="2.329404771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:08.246603641 +0000 UTC m=+505.295713456" watchObservedRunningTime="2026-11-01 20:39:08.247676868 +0000 UTC m=+505.296786687"
Nov 01 20:39:08.414538 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.414516    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:08.464897 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.464864    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:08.782066 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:08.782041    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"]
Nov 01 20:39:08.784019 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:08.783990    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfc3889_952c_448a_98ef_8098113fa769.slice/crio-c16f4d9ec30ef26b885a8bf891546d874795b98826d3014abdfe78182e5f9ddc WatchSource:0}: Error finding container c16f4d9ec30ef26b885a8bf891546d874795b98826d3014abdfe78182e5f9ddc: Status 404 returned error can't find the container with id c16f4d9ec30ef26b885a8bf891546d874795b98826d3014abdfe78182e5f9ddc
Nov 01 20:39:09.136735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:09.136676    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6" event={"ID":"7cfc3889-952c-448a-98ef-8098113fa769","Type":"ContainerStarted","Data":"c16f4d9ec30ef26b885a8bf891546d874795b98826d3014abdfe78182e5f9ddc"}
Nov 01 20:39:10.141510 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.141473    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6" event={"ID":"7cfc3889-952c-448a-98ef-8098113fa769","Type":"ContainerStarted","Data":"fe88eb26c9da9e021cdc2be333961282366d88ca48459d287dd785f3d60c910c"}
Nov 01 20:39:10.259459 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.259414    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6" podStartSLOduration=2.540100006 podStartE2EDuration="3.259400615s" totalImagesPullingTime="719.300609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:10.257795206 +0000 UTC m=+507.306905025" watchObservedRunningTime="2026-11-01 20:39:10.259400615 +0000 UTC m=+507.308510516"
Nov 01 20:39:10.414906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.414884    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:10.878412 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.878337    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"]
Nov 01 20:39:10.881619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.881603    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:10.985042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.985012    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 20:39:10.986387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.986368    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"]
Nov 01 20:39:10.996266 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.996236    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rrw\" (UniqueName: \"kubernetes.io/projected/5d760d74-afb7-4acf-896c-7afc9099a392-kube-api-access-56rrw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:10.996373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.996275    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:10.996373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.996301    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:10.996449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.996371    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:10.996449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.996406    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d760d74-afb7-4acf-896c-7afc9099a392-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:10.996520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:10.996483    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.096897 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.096873    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rrw\" (UniqueName: \"kubernetes.io/projected/5d760d74-afb7-4acf-896c-7afc9099a392-kube-api-access-56rrw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097001 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.096909    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097001 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.096927    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097199 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.097180    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.097218    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d760d74-afb7-4acf-896c-7afc9099a392-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.097241    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097435 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.097412    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097521 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.097501    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.097576 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.097532    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.098950 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.098931    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d760d74-afb7-4acf-896c-7afc9099a392-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.099389 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.099372    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d760d74-afb7-4acf-896c-7afc9099a392-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.154577 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.154550    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rrw\" (UniqueName: \"kubernetes.io/projected/5d760d74-afb7-4acf-896c-7afc9099a392-kube-api-access-56rrw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rgff6\" (UID: \"5d760d74-afb7-4acf-896c-7afc9099a392\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.193351 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.193333    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:11.559015 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:11.558952    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"]
Nov 01 20:39:11.561871 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:11.561847    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d760d74_afb7_4acf_896c_7afc9099a392.slice/crio-be57731783d804013fc2e4ee30f02fb96434fee4b9a38402d2121e443965d346 WatchSource:0}: Error finding container be57731783d804013fc2e4ee30f02fb96434fee4b9a38402d2121e443965d346: Status 404 returned error can't find the container with id be57731783d804013fc2e4ee30f02fb96434fee4b9a38402d2121e443965d346
Nov 01 20:39:12.150749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:12.150720    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6" event={"ID":"5d760d74-afb7-4acf-896c-7afc9099a392","Type":"ContainerStarted","Data":"be57731783d804013fc2e4ee30f02fb96434fee4b9a38402d2121e443965d346"}
Nov 01 20:39:13.155429 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:13.155391    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6" event={"ID":"5d760d74-afb7-4acf-896c-7afc9099a392","Type":"ContainerStarted","Data":"f277107e566336918c6bb776350a0011c7822ffb304847289f17180ce7a83138"}
Nov 01 20:39:13.194164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:13.194142    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:13.270533 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:13.270489    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6" podStartSLOduration=2.57460471 podStartE2EDuration="3.270476125s" totalImagesPullingTime="695.871415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:13.270074744 +0000 UTC m=+510.319184564" watchObservedRunningTime="2026-11-01 20:39:13.270476125 +0000 UTC m=+510.319585942"
Nov 01 20:39:14.464132 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.464102    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:14.475859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.475835    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:14.740290 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.740231    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"]
Nov 01 20:39:14.743467 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.743451    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:14.799711 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.799685    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 20:39:14.801187 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.801164    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"]
Nov 01 20:39:14.926974 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.926953    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:14.927096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.926980    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:14.927096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.927006    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:14.927096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.927069    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a653fd-4a69-48ab-a68b-ffc0011441cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:14.927211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.927128    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:14.927211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:14.927156    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4rs\" (UniqueName: \"kubernetes.io/projected/f3a653fd-4a69-48ab-a68b-ffc0011441cf-kube-api-access-6f4rs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028369    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028399    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028423    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028621 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028553    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a653fd-4a69-48ab-a68b-ffc0011441cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028621 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028589    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028776 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028756    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4rs\" (UniqueName: \"kubernetes.io/projected/f3a653fd-4a69-48ab-a68b-ffc0011441cf-kube-api-access-6f4rs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028829 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028777    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028861    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.028886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.028875    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.030602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.030584    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a653fd-4a69-48ab-a68b-ffc0011441cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.030921 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.030901    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a653fd-4a69-48ab-a68b-ffc0011441cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.085818 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.085794    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4rs\" (UniqueName: \"kubernetes.io/projected/f3a653fd-4a69-48ab-a68b-ffc0011441cf-kube-api-access-6f4rs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-86g27\" (UID: \"f3a653fd-4a69-48ab-a68b-ffc0011441cf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.173430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.173407    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv"
Nov 01 20:39:15.355800 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.355730    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:15.724296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:15.724171    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"]
Nov 01 20:39:15.726849 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:15.726826    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a653fd_4a69_48ab_a68b_ffc0011441cf.slice/crio-db497c0fb47a14f3a728fee29e9818830d2662cf410db63021462e92b16fc5c8 WatchSource:0}: Error finding container db497c0fb47a14f3a728fee29e9818830d2662cf410db63021462e92b16fc5c8: Status 404 returned error can't find the container with id db497c0fb47a14f3a728fee29e9818830d2662cf410db63021462e92b16fc5c8
Nov 01 20:39:16.167942 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:16.167909    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27" event={"ID":"f3a653fd-4a69-48ab-a68b-ffc0011441cf","Type":"ContainerStarted","Data":"db497c0fb47a14f3a728fee29e9818830d2662cf410db63021462e92b16fc5c8"}
Nov 01 20:39:17.173760 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:17.173722    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27" event={"ID":"f3a653fd-4a69-48ab-a68b-ffc0011441cf","Type":"ContainerStarted","Data":"5cbb1b281f3067a60e3c3d673d3a2c46170fb72cd4e8f8be399da094bcfbdbdb"}
Nov 01 20:39:17.292565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:17.292503    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27" podStartSLOduration=2.559685388 podStartE2EDuration="3.292489954s" totalImagesPullingTime="732.804566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:17.290986293 +0000 UTC m=+514.340096111" watchObservedRunningTime="2026-11-01 20:39:17.292489954 +0000 UTC m=+514.341599771"
Nov 01 20:39:17.356336 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:17.356305    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:18.415188 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:18.415160    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:18.427258 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:18.427237    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:19.193715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:19.193690    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-t4dq6"
Nov 01 20:39:21.194415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.194383    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:21.206790 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.206762    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:21.565544 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.565469    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6xbbn"]
Nov 01 20:39:21.569064 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.569046    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.580254 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.580234    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-home\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.580372 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.580260    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1920fad9-b098-4cd7-8786-62261b1bb34b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.580372 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.580337    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.580492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.580408    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.580492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.580450    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdjt\" (UniqueName: \"kubernetes.io/projected/1920fad9-b098-4cd7-8786-62261b1bb34b-kube-api-access-zbdjt\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.580581 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.580532    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.624179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.624149    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 20:39:21.625738 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.625713    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6xbbn"]
Nov 01 20:39:21.681699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.681644    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-home\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.681699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.681700    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1920fad9-b098-4cd7-8786-62261b1bb34b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.681915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.681760    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.681915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.681797    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.681915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.681829    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdjt\" (UniqueName: \"kubernetes.io/projected/1920fad9-b098-4cd7-8786-62261b1bb34b-kube-api-access-zbdjt\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.681915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.681874    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.682524 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.682473    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.682685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.682567    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-home\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.682838 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.682814    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.688705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.685081    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1920fad9-b098-4cd7-8786-62261b1bb34b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.688705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.685360    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1920fad9-b098-4cd7-8786-62261b1bb34b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.740447 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.740419    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdjt\" (UniqueName: \"kubernetes.io/projected/1920fad9-b098-4cd7-8786-62261b1bb34b-kube-api-access-zbdjt\") pod \"echo-simulator-kserve-64b6f8f96c-6xbbn\" (UID: \"1920fad9-b098-4cd7-8786-62261b1bb34b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:21.880944 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:21.880889    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:22.204322 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:22.204294    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rgff6"
Nov 01 20:39:22.246229 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:22.246202    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6xbbn"]
Nov 01 20:39:22.249100 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:22.249076    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1920fad9_b098_4cd7_8786_62261b1bb34b.slice/crio-f1b053ab16da10342d6db1ebcd4f8a751ab31086925579156b2d4f1c1287e991 WatchSource:0}: Error finding container f1b053ab16da10342d6db1ebcd4f8a751ab31086925579156b2d4f1c1287e991: Status 404 returned error can't find the container with id f1b053ab16da10342d6db1ebcd4f8a751ab31086925579156b2d4f1c1287e991
Nov 01 20:39:23.199932 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:23.199898    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn" event={"ID":"1920fad9-b098-4cd7-8786-62261b1bb34b","Type":"ContainerStarted","Data":"f1b053ab16da10342d6db1ebcd4f8a751ab31086925579156b2d4f1c1287e991"}
Nov 01 20:39:23.364075 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:39:23.364040    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:39:25.209528 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:25.209496    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn" event={"ID":"1920fad9-b098-4cd7-8786-62261b1bb34b","Type":"ContainerStarted","Data":"31be84dfbf1ce6144805a01919a213c2dfa387d3bd76c516786105369d74abb0"}
Nov 01 20:39:25.324473 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:25.324423    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn" podStartSLOduration=1.7079426 podStartE2EDuration="4.32440781s" totalImagesPullingTime="2.61646521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:25.323968965 +0000 UTC m=+522.373078781" watchObservedRunningTime="2026-11-01 20:39:25.32440781 +0000 UTC m=+522.373517628"
Nov 01 20:39:25.356250 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:25.356225    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:25.368344 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:25.368325    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:25.881370 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:25.881342    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:26.228219 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:26.228194    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-86g27"
Nov 01 20:39:29.000059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.000028    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"]
Nov 01 20:39:29.003495 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.003471    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.038205 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.038180    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.038306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.038216    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.038306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.038251    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.038441 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.038415    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt64d\" (UniqueName: \"kubernetes.io/projected/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-kube-api-access-dt64d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.038486 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.038475    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.038544 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.038528    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.060321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.060300    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"]
Nov 01 20:39:29.109298 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.109279    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 20:39:29.139626 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.139599    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.139741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.139635    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.139741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.139706    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.139741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.139735    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.139901 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.139776    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.139901 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.139835    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt64d\" (UniqueName: \"kubernetes.io/projected/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-kube-api-access-dt64d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.140106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.140083    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.140106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.140094    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.140218 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.140125    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.141764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.141744    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.142033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.142016    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.197075 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.197045    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt64d\" (UniqueName: \"kubernetes.io/projected/e2403ce9-732c-42f0-b2d4-eb1fb462e1d6-kube-api-access-dt64d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm\" (UID: \"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.327769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.327706    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:29.695333 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:29.695300    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"]
Nov 01 20:39:29.697438 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:29.697401    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2403ce9_732c_42f0_b2d4_eb1fb462e1d6.slice/crio-6b4e6499702132a89d494e0a8e7701e8012349d9d1c5507ee706aef2680a2dbe WatchSource:0}: Error finding container 6b4e6499702132a89d494e0a8e7701e8012349d9d1c5507ee706aef2680a2dbe: Status 404 returned error can't find the container with id 6b4e6499702132a89d494e0a8e7701e8012349d9d1c5507ee706aef2680a2dbe
Nov 01 20:39:30.236125 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:30.236086    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" event={"ID":"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6","Type":"ContainerStarted","Data":"6b4e6499702132a89d494e0a8e7701e8012349d9d1c5507ee706aef2680a2dbe"}
Nov 01 20:39:31.241564 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:31.241529    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" event={"ID":"e2403ce9-732c-42f0-b2d4-eb1fb462e1d6","Type":"ContainerStarted","Data":"fdfaef082d6eb6d9b68b7ee8a074b6c33b7fe54333ce9d47e513d63d95c09d02"}
Nov 01 20:39:31.328390 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:31.328362    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:31.358778 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:31.358713    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podStartSLOduration=2.647892557 podStartE2EDuration="3.358696459s" totalImagesPullingTime="710.803902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:31.356724182 +0000 UTC m=+528.405833999" watchObservedRunningTime="2026-11-01 20:39:31.358696459 +0000 UTC m=+528.407806281"
Nov 01 20:39:31.881534 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:31.881505    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:31.893990 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:31.893962    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:32.256767 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:32.256742    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6xbbn"
Nov 01 20:39:39.328577 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:39.328549    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:39.340569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:39.340541    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:40.288690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:40.288642    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:39:49.454461 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.454425    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"]
Nov 01 20:39:49.459842 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.459823    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504426    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504456    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkwl\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504476    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504493    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504548    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504581    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504604    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504634    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.504790 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.504650    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.516085 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.516065    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\""
Nov 01 20:39:49.565933 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.565904    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 20:39:49.567950 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.567925    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"]
Nov 01 20:39:49.605757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605725    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605757    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkwl\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605780    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605795    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605810    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605832    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605856    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605880    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.605978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.605904    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.606315 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.606223    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.606315 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.606273    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.606483 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.606456    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.606611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.606522    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.606886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.606866    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.608108 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.608089    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.608394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.608370    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.662921 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.662874    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.663043 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.662965    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkwl\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl\") pod \"maas-default-gateway-openshift-default-58c6585447-ppfz4\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:49.772258 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:49.772197    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:50.244496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:50.244464    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"]
Nov 01 20:39:50.285995 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:39:50.285960    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada7e173_1bca_4229_becb_69e9edc1250f.slice/crio-e055c52472a4d513fc77fab7ea93b9e5a60d84852eb0b530dcd23bf18517c082 WatchSource:0}: Error finding container e055c52472a4d513fc77fab7ea93b9e5a60d84852eb0b530dcd23bf18517c082: Status 404 returned error can't find the container with id e055c52472a4d513fc77fab7ea93b9e5a60d84852eb0b530dcd23bf18517c082
Nov 01 20:39:50.320338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:50.320311    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" event={"ID":"ada7e173-1bca-4229-becb-69e9edc1250f","Type":"ContainerStarted","Data":"e055c52472a4d513fc77fab7ea93b9e5a60d84852eb0b530dcd23bf18517c082"}
Nov 01 20:39:53.373698 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:39:53.373647    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:39:53.969202 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:53.969172    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:39:53.969307 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:53.969238    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:39:53.969307 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:53.969266    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:39:54.346096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:54.346003    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" event={"ID":"ada7e173-1bca-4229-becb-69e9edc1250f","Type":"ContainerStarted","Data":"27f46f3be09257d3fd46767d380297de8d578a20bdceea32f048f7104639c2e4"}
Nov 01 20:39:54.519392 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:54.519333    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podStartSLOduration=1.8387692 podStartE2EDuration="5.519318116s" totalImagesPullingTime="3.680548916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:54.51717523 +0000 UTC m=+551.566285052" watchObservedRunningTime="2026-11-01 20:39:54.519318116 +0000 UTC m=+551.568427937"
Nov 01 20:39:54.772705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:54.772665    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:39:55.774420 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:55.774388    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ppfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 01 20:39:55.774850 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:55.774452    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 01 20:39:56.772579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:56.772542    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ppfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 01 20:39:56.772770 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:56.772597    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 01 20:39:57.773210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:57.773175    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ppfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 01 20:39:57.773610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:57.773259    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 01 20:39:58.772748 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:58.772711    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ppfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 01 20:39:58.772925 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:58.772776    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 01 20:39:59.545705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.545675    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"]
Nov 01 20:39:59.549112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.549092    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.619714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.609229    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"]
Nov 01 20:39:59.619714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.619375    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2nr\" (UniqueName: \"kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr\") pod \"ai-gateway-controller-5f6dd4f69f-kkb9h\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.619714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.619459    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-kkb9h\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.656732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.656702    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 01 20:39:59.656732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.656724    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 01 20:39:59.656933 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.656757    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-tshhz\""
Nov 01 20:39:59.720431 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.720399    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2nr\" (UniqueName: \"kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr\") pod \"ai-gateway-controller-5f6dd4f69f-kkb9h\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.720603 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.720450    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-kkb9h\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.722880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.722851    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-kkb9h\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.772690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.772648    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ppfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 01 20:39:59.772852 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.772720    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 01 20:39:59.778284 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.778257    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2nr\" (UniqueName: \"kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr\") pod \"ai-gateway-controller-5f6dd4f69f-kkb9h\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:39:59.860836 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:39:59.860753    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:40:00.235681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:00.235602    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"]
Nov 01 20:40:00.372338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:00.372302    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h" event={"ID":"87385df9-fbe1-4994-bceb-0ead5be0a4c5","Type":"ContainerStarted","Data":"0deb3e3d3c5674dfb4109f17961be44c895afe0063943d57b475f1f26e4c8e75"}
Nov 01 20:40:00.772504 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:00.772466    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ppfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 01 20:40:00.772972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:00.772533    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 01 20:40:01.990078 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:01.990041    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:40:01.990523 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:01.990412    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:40:01.991300 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:01.991277    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:40:04.528347 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.528317    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"]
Nov 01 20:40:04.531764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.531745    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669051 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669021    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669051 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669053    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669227 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669079    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669227 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669204    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669299 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669231    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669299 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669249    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95l8\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669356    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669407 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669378    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.669463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.669451    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.732967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.732944    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"]
Nov 01 20:40:04.770369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770342    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.770496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770382    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.770496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770412    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.770496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770435    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95l8\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.770496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770480    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.770688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770504    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.770919 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.770545    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.771168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.771137    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.771321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.771296    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.771606 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.771476    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.773048 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.773018    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.773444 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.773294    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.773747 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.773722    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.774079 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.774056    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.774473 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.774445    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.774617 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.774595    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.828188 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.828141    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95l8\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.829724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.829696    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2zkzj\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:04.844330 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:04.844312    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:05.264974 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.264953    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"]
Nov 01 20:40:05.267378 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:40:05.267352    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d493d13_62e2_4a95_bab3_d51702727c32.slice/crio-68cf9aa8663bee8f2a3f29266036f0fe7906921b4b0351556aa40ece0cce3dc4 WatchSource:0}: Error finding container 68cf9aa8663bee8f2a3f29266036f0fe7906921b4b0351556aa40ece0cce3dc4: Status 404 returned error can't find the container with id 68cf9aa8663bee8f2a3f29266036f0fe7906921b4b0351556aa40ece0cce3dc4
Nov 01 20:40:05.270274 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.270241    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:40:05.270350 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.270302    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:40:05.270350 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.270327    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:40:05.401960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.401921    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" event={"ID":"1d493d13-62e2-4a95-bab3-d51702727c32","Type":"ContainerStarted","Data":"48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396"}
Nov 01 20:40:05.402119 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.402003    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" event={"ID":"1d493d13-62e2-4a95-bab3-d51702727c32","Type":"ContainerStarted","Data":"68cf9aa8663bee8f2a3f29266036f0fe7906921b4b0351556aa40ece0cce3dc4"}
Nov 01 20:40:05.520365 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.520275    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podStartSLOduration=1.520260545 podStartE2EDuration="1.520260545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:05.518726429 +0000 UTC m=+562.567836247" watchObservedRunningTime="2026-11-01 20:40:05.520260545 +0000 UTC m=+562.569370363"
Nov 01 20:40:05.844744 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:05.844640    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:06.846486 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:06.846351    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:06.846486 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:06.846404    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:07.428391 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:07.428348    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h" event={"ID":"87385df9-fbe1-4994-bceb-0ead5be0a4c5","Type":"ContainerStarted","Data":"75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892"}
Nov 01 20:40:07.428625 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:07.428608    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:40:07.544346 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:07.544296    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h" podStartSLOduration=2.234804535 podStartE2EDuration="8.54428205s" totalImagesPullingTime="6.309477515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:07.543250721 +0000 UTC m=+564.592360538" watchObservedRunningTime="2026-11-01 20:40:07.54428205 +0000 UTC m=+564.593391867"
Nov 01 20:40:07.845725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:07.845615    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:07.845725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:07.845700    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:08.845011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:08.844976    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:08.845402 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:08.845033    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:09.716585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.716535    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"]
Nov 01 20:40:09.732624 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.732592    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:09.775776 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.775742    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"]
Nov 01 20:40:09.825005 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.824973    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 20:40:09.825259 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.825057    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m2pkv\""
Nov 01 20:40:09.844077 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.844038    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd95e301-fe56-462e-9a29-f83709bb22b5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jdsm2\" (UID: \"fd95e301-fe56-462e-9a29-f83709bb22b5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:09.844229 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.844182    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hmr\" (UniqueName: \"kubernetes.io/projected/fd95e301-fe56-462e-9a29-f83709bb22b5-kube-api-access-x9hmr\") pod \"payload-pre-processing-7b98bb678c-jdsm2\" (UID: \"fd95e301-fe56-462e-9a29-f83709bb22b5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:09.844902 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.844878    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:09.845011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.844926    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:09.861956 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.861926    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"]
Nov 01 20:40:09.865643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.865625    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:09.922944 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.922846    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"]
Nov 01 20:40:09.945566 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.945530    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd95e301-fe56-462e-9a29-f83709bb22b5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jdsm2\" (UID: \"fd95e301-fe56-462e-9a29-f83709bb22b5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:09.945767 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.945622    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hmr\" (UniqueName: \"kubernetes.io/projected/fd95e301-fe56-462e-9a29-f83709bb22b5-kube-api-access-x9hmr\") pod \"payload-pre-processing-7b98bb678c-jdsm2\" (UID: \"fd95e301-fe56-462e-9a29-f83709bb22b5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:09.946296 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:09.946272    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd95e301-fe56-462e-9a29-f83709bb22b5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jdsm2\" (UID: \"fd95e301-fe56-462e-9a29-f83709bb22b5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:10.021374 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.021294    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hmr\" (UniqueName: \"kubernetes.io/projected/fd95e301-fe56-462e-9a29-f83709bb22b5-kube-api-access-x9hmr\") pod \"payload-pre-processing-7b98bb678c-jdsm2\" (UID: \"fd95e301-fe56-462e-9a29-f83709bb22b5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:10.045011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.044981    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:10.046320 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.046297    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkfl\" (UniqueName: \"kubernetes.io/projected/fa5c0b0a-df95-4568-a3b5-ffdb6ace8352-kube-api-access-cmkfl\") pod \"payload-processing-55fbbbf9dc-mp5sx\" (UID: \"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.046428 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.046348    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5c0b0a-df95-4568-a3b5-ffdb6ace8352-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-mp5sx\" (UID: \"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.148267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.146950    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5c0b0a-df95-4568-a3b5-ffdb6ace8352-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-mp5sx\" (UID: \"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.148267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.147052    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkfl\" (UniqueName: \"kubernetes.io/projected/fa5c0b0a-df95-4568-a3b5-ffdb6ace8352-kube-api-access-cmkfl\") pod \"payload-processing-55fbbbf9dc-mp5sx\" (UID: \"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.148267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.148029    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5c0b0a-df95-4568-a3b5-ffdb6ace8352-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-mp5sx\" (UID: \"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.225683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.225591    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkfl\" (UniqueName: \"kubernetes.io/projected/fa5c0b0a-df95-4568-a3b5-ffdb6ace8352-kube-api-access-cmkfl\") pod \"payload-processing-55fbbbf9dc-mp5sx\" (UID: \"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.444085 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.444058    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"]
Nov 01 20:40:10.478549 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.478515    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:10.845477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.845370    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:10.845477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.845434    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:10.880956 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:10.880930    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"]
Nov 01 20:40:10.883788 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:40:10.883760    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5c0b0a_df95_4568_a3b5_ffdb6ace8352.slice/crio-5f592298aa786e8cec9044600dd5eec25581b1133337cdbed48a9da6e714543d WatchSource:0}: Error finding container 5f592298aa786e8cec9044600dd5eec25581b1133337cdbed48a9da6e714543d: Status 404 returned error can't find the container with id 5f592298aa786e8cec9044600dd5eec25581b1133337cdbed48a9da6e714543d
Nov 01 20:40:11.450823 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:11.450784    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx" event={"ID":"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352","Type":"ContainerStarted","Data":"5f592298aa786e8cec9044600dd5eec25581b1133337cdbed48a9da6e714543d"}
Nov 01 20:40:11.452279 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:11.452242    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2" event={"ID":"fd95e301-fe56-462e-9a29-f83709bb22b5","Type":"ContainerStarted","Data":"3dbec51a543f3c607a8af7e91195d92d24392d1ca29684af71388a80a40c2d3d"}
Nov 01 20:40:11.845210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:11.844704    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:11.845210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:11.844778    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:12.844915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:12.844867    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2zkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 01 20:40:12.845421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:12.844931    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 01 20:40:13.978115 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:13.978079    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:13.978516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:13.978426    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:13.979219 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:13.979198    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:40:14.474794 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:14.474763    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx" event={"ID":"fa5c0b0a-df95-4568-a3b5-ffdb6ace8352","Type":"ContainerStarted","Data":"3f99b8a91979803aa5b07850ca909e48967fc7e2e0cab333bfe908bbbcd0a5fd"}
Nov 01 20:40:14.474994 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:14.474867    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:14.476583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:14.476554    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2" event={"ID":"fd95e301-fe56-462e-9a29-f83709bb22b5","Type":"ContainerStarted","Data":"1048b8285e9374587794d18940a85ea53ce6ab8fa413cbc5e20a023e4c689527"}
Nov 01 20:40:14.476843 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:14.476828    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:14.590468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:14.590414    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx" podStartSLOduration=2.6503134360000002 podStartE2EDuration="5.590400849s" totalImagesPullingTime="2.940087413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:14.58906953 +0000 UTC m=+571.638179349" watchObservedRunningTime="2026-11-01 20:40:14.590400849 +0000 UTC m=+571.639510666"
Nov 01 20:40:14.706737 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:14.706635    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2" podStartSLOduration=2.063804202 podStartE2EDuration="5.706620368s" totalImagesPullingTime="3.642816166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:14.705191445 +0000 UTC m=+571.754301263" watchObservedRunningTime="2026-11-01 20:40:14.706620368 +0000 UTC m=+571.755730180"
Nov 01 20:40:18.434892 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:18.434858    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:40:18.825837 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:18.825751    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7f499969f4-rlkg6"]
Nov 01 20:40:18.829406 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:18.829389    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:18.885037 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:18.885010    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f499969f4-rlkg6"]
Nov 01 20:40:18.951006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:18.950974    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0665b-f370-478a-a7d2-abcc6021bfa8-metrics-certs\") pod \"ai-gateway-controller-7f499969f4-rlkg6\" (UID: \"e7f0665b-f370-478a-a7d2-abcc6021bfa8\") " pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:18.951179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:18.951041    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mv2b\" (UniqueName: \"kubernetes.io/projected/e7f0665b-f370-478a-a7d2-abcc6021bfa8-kube-api-access-2mv2b\") pod \"ai-gateway-controller-7f499969f4-rlkg6\" (UID: \"e7f0665b-f370-478a-a7d2-abcc6021bfa8\") " pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:19.052465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:19.052438    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mv2b\" (UniqueName: \"kubernetes.io/projected/e7f0665b-f370-478a-a7d2-abcc6021bfa8-kube-api-access-2mv2b\") pod \"ai-gateway-controller-7f499969f4-rlkg6\" (UID: \"e7f0665b-f370-478a-a7d2-abcc6021bfa8\") " pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:19.052612 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:19.052494    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0665b-f370-478a-a7d2-abcc6021bfa8-metrics-certs\") pod \"ai-gateway-controller-7f499969f4-rlkg6\" (UID: \"e7f0665b-f370-478a-a7d2-abcc6021bfa8\") " pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:19.054806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:19.054782    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0665b-f370-478a-a7d2-abcc6021bfa8-metrics-certs\") pod \"ai-gateway-controller-7f499969f4-rlkg6\" (UID: \"e7f0665b-f370-478a-a7d2-abcc6021bfa8\") " pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:19.110192 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:19.110130    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mv2b\" (UniqueName: \"kubernetes.io/projected/e7f0665b-f370-478a-a7d2-abcc6021bfa8-kube-api-access-2mv2b\") pod \"ai-gateway-controller-7f499969f4-rlkg6\" (UID: \"e7f0665b-f370-478a-a7d2-abcc6021bfa8\") " pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:19.141053 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:19.141017    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:19.513554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:19.513527    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f499969f4-rlkg6"]
Nov 01 20:40:19.515754 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:40:19.515728    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f0665b_f370_478a_a7d2_abcc6021bfa8.slice/crio-266db888c6e5f99004f3c767b102e9fb3cfd39653142aea60f12ef40d605c848 WatchSource:0}: Error finding container 266db888c6e5f99004f3c767b102e9fb3cfd39653142aea60f12ef40d605c848: Status 404 returned error can't find the container with id 266db888c6e5f99004f3c767b102e9fb3cfd39653142aea60f12ef40d605c848
Nov 01 20:40:20.504019 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:20.503977    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6" event={"ID":"e7f0665b-f370-478a-a7d2-abcc6021bfa8","Type":"ContainerStarted","Data":"fe12cca456e8333b218d344c59bbd381a5781a12cc60eb430cb664edd1dcc390"}
Nov 01 20:40:20.504210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:20.504044    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6" event={"ID":"e7f0665b-f370-478a-a7d2-abcc6021bfa8","Type":"ContainerStarted","Data":"266db888c6e5f99004f3c767b102e9fb3cfd39653142aea60f12ef40d605c848"}
Nov 01 20:40:20.504210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:20.504188    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:20.620180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:20.620084    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6" podStartSLOduration=1.818662607 podStartE2EDuration="2.620070123s" totalImagesPullingTime="801.407516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:20.618284213 +0000 UTC m=+577.667394028" watchObservedRunningTime="2026-11-01 20:40:20.620070123 +0000 UTC m=+577.669179941"
Nov 01 20:40:23.363452 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:40:23.363416    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:40:25.484998 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:25.484922    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-mp5sx"
Nov 01 20:40:25.484998 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:25.484988    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jdsm2"
Nov 01 20:40:31.510807 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:31.510769    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7f499969f4-rlkg6"
Nov 01 20:40:31.678477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:31.678444    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"]
Nov 01 20:40:31.678707 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:31.678673    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h" podUID="87385df9-fbe1-4994-bceb-0ead5be0a4c5" containerName="manager" containerID="cri-o://75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892" gracePeriod=30
Nov 01 20:40:31.923344 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:31.923324    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:40:32.077178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.077102    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs\") pod \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") "
Nov 01 20:40:32.077308 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.077190    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr\" (UniqueName: \"kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr\") pod \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\" (UID: \"87385df9-fbe1-4994-bceb-0ead5be0a4c5\") "
Nov 01 20:40:32.079228 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.079199    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs" pod "87385df9-fbe1-4994-bceb-0ead5be0a4c5" (UID: "87385df9-fbe1-4994-bceb-0ead5be0a4c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:32.079228 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.079200    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr" pod "87385df9-fbe1-4994-bceb-0ead5be0a4c5" (UID: "87385df9-fbe1-4994-bceb-0ead5be0a4c5"). InnerVolumeSpecName "kube-api-access-xf2nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:32.177851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.177818    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87385df9-fbe1-4994-bceb-0ead5be0a4c5-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:32.177851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.177843    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2nr\" (UniqueName: \"kubernetes.io/projected/87385df9-fbe1-4994-bceb-0ead5be0a4c5-kube-api-access-xf2nr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:32.557531 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.557492    4872 generic.go:356] "Generic (PLEG): container finished" podID="87385df9-fbe1-4994-bceb-0ead5be0a4c5" containerID="75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892" exitCode=0
Nov 01 20:40:32.557927 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.557537    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"
Nov 01 20:40:32.557927 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.557576    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h" event={"ID":"87385df9-fbe1-4994-bceb-0ead5be0a4c5","Type":"ContainerDied","Data":"75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892"}
Nov 01 20:40:32.557927 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.557628    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h" event={"ID":"87385df9-fbe1-4994-bceb-0ead5be0a4c5","Type":"ContainerDied","Data":"0deb3e3d3c5674dfb4109f17961be44c895afe0063943d57b475f1f26e4c8e75"}
Nov 01 20:40:32.557927 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.557649    4872 scope.go:122] "RemoveContainer" containerID="75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892"
Nov 01 20:40:32.567399 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.567377    4872 scope.go:122] "RemoveContainer" containerID="75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892"
Nov 01 20:40:32.567634 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:40:32.567614    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892\": container with ID starting with 75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892 not found: ID does not exist" containerID="75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892"
Nov 01 20:40:32.567727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.567642    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892"} err="failed to get container status \"75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892\": rpc error: code = NotFound desc = could not find container \"75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892\": container with ID starting with 75c5a774db155cb4305f9c32b20edf858460f99e5cd075ff091ac90884416892 not found: ID does not exist"
Nov 01 20:40:32.722585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.722550    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"]
Nov 01 20:40:32.773148 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:32.773109    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-kkb9h"]
Nov 01 20:40:33.077316 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:33.077285    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87385df9-fbe1-4994-bceb-0ead5be0a4c5" path="/var/lib/kubelet/pods/87385df9-fbe1-4994-bceb-0ead5be0a4c5/volumes"
Nov 01 20:40:34.749757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.749726    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"]
Nov 01 20:40:34.750120 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.750102    4872 cpu_manager.go:414] "containerMap: removing container" podUID="87385df9-fbe1-4994-bceb-0ead5be0a4c5" containerName="manager"
Nov 01 20:40:34.750120 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.750112    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87385df9-fbe1-4994-bceb-0ead5be0a4c5" containerName="manager"
Nov 01 20:40:34.750197 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.750187    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="87385df9-fbe1-4994-bceb-0ead5be0a4c5" containerName="manager"
Nov 01 20:40:34.754707 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.754691    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.860714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.860686    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"]
Nov 01 20:40:34.901876 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.901845    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.901998 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.901888    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.901998 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.901915    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.901998 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.901957    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.902123 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.902073    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.902123 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.902104    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.902258 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.902243    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.902316 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.902278    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:34.902316 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:34.902302    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlrc\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.003715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003611    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.003715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003651    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.003920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003738    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlrc\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.003920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003780    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.003920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003808    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.003920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003828    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003914    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.003986    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.004010    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.004236    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.004389    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.004435    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.004704    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.004806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.004791    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.006477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.006454    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.006710 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.006690    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.061540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.061501    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.061693 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.061628    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlrc\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc\") pod \"maas-default-gateway-openshift-default-58c6585447-8jpjl\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.069495 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.069474    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:35.494489 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.494382    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"]
Nov 01 20:40:35.496686 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:40:35.496635    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fa36eb_539d_498a_8bb6_31f6982cc240.slice/crio-86cab3e00341b5bf415a64dc5337003a299f46f64628b881c0d5dd4244c97391 WatchSource:0}: Error finding container 86cab3e00341b5bf415a64dc5337003a299f46f64628b881c0d5dd4244c97391: Status 404 returned error can't find the container with id 86cab3e00341b5bf415a64dc5337003a299f46f64628b881c0d5dd4244c97391
Nov 01 20:40:35.498919 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.498893    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:40:35.498976 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.498952    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:40:35.499018 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.498981    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:40:35.574447 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.574409    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" event={"ID":"47fa36eb-539d-498a-8bb6-31f6982cc240","Type":"ContainerStarted","Data":"2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda"}
Nov 01 20:40:35.574583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.574475    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" event={"ID":"47fa36eb-539d-498a-8bb6-31f6982cc240","Type":"ContainerStarted","Data":"86cab3e00341b5bf415a64dc5337003a299f46f64628b881c0d5dd4244c97391"}
Nov 01 20:40:35.691782 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:35.691713    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podStartSLOduration=1.691694578 podStartE2EDuration="1.691694578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:35.691027292 +0000 UTC m=+592.740137114" watchObservedRunningTime="2026-11-01 20:40:35.691694578 +0000 UTC m=+592.740804397"
Nov 01 20:40:36.069825 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:36.069794    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:36.071409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:36.071382    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:36.071534 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:36.071432    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:37.070335 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:37.070300    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:37.070825 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:37.070348    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:38.070630 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:38.070597    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:38.071043 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:38.070648    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:39.070145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:39.070114    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:39.070333 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:39.070167    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:40.070640 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:40.070603    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:40.071074 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:40.070690    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:41.070498 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:41.070465    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:41.070686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:41.070526    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:42.070978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:42.070941    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8jpjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:40:42.071378 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:42.071008    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 20:40:43.130669 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.130622    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:40:43.131449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.131427    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:40:43.137396 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.137378    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:40:43.137807 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.137788    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:40:43.485977 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.485943    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:43.486338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.486318    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:43.487076 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:40:43.487053    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:40:53.366998 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:40:53.366959    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:41:05.009510 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.009473    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"]
Nov 01 20:41:05.012852 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.012837    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.077519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.077489    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-dd038b-openshift-default-dockercfg-vkx72\""
Nov 01 20:41:05.107624 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.107591    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"]
Nov 01 20:41:05.113295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113266    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113440 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113323    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113490 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113439    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113490 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113479    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113568 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113507    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113568 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113554    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113756 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113730    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113857 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113831    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.113954 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.113893    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559qk\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.214755 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.214717    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.214944 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.214822    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215204 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215182    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559qk\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215309 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215246    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215393 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215372    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215460 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215445    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215515 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215473    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215571 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215519    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215571 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215555    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.215704 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215607    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.216056 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.215971    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.216056 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.216005    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.216210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.216149    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.216335 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.216318    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.217469 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.217449    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.217816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.217799    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.273686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.273581    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.273967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.273948    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559qk\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.325483 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.325454    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:05.750002 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.749979    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"]
Nov 01 20:41:05.752155 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:05.752128    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d37b7fd_beb4_428c_8f6b_ca28097b15a9.slice/crio-48099fbffa332b9cd0df625f9b7dc4411b7805fa4181037166727630af2e95e7 WatchSource:0}: Error finding container 48099fbffa332b9cd0df625f9b7dc4411b7805fa4181037166727630af2e95e7: Status 404 returned error can't find the container with id 48099fbffa332b9cd0df625f9b7dc4411b7805fa4181037166727630af2e95e7
Nov 01 20:41:05.754198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.754170    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:05.754260 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.754226    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:05.754301 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:05.754257    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:06.010273 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.010199    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"]
Nov 01 20:41:06.018796 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.018768    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.072702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.072640    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9ae21-openshift-default-dockercfg-k8hz9\""
Nov 01 20:41:06.072886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.072860    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"]
Nov 01 20:41:06.123297 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123250    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123476 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123313    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbg9\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123542 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123478    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123542 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123526    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123624 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123578    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123696 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123677    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123734    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123771    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.123876 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.123857    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225114 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225070    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225124    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225155    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225198    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225224    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225253    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225302    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225348    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225388    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbg9\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.225832 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.225811    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.226074 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.226052    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.226254 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.226189    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.226369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.226248    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.226491 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.226353    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.228424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.228396    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.228547 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.228529    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.283325 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.283268    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbg9\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.283554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.283533    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.332362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.332309    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:06.334573 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.334540    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"]
Nov 01 20:41:06.338189 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.338169    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.405052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.405017    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1a1e9e-openshift-default-dockercfg-xvvfj\""
Nov 01 20:41:06.407750 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.407724    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"]
Nov 01 20:41:06.427383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427354    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427398    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwk2\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427562    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427631    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427756 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427737    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427810 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427781    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427900 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427880    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427932    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.427960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.427954    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.528752 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528720    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.528896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528779    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.528896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528816    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.528896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528849    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528925    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528958    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.528993    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529029    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529130    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwk2\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529273 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529251    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529310    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529347    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529512 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529485    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.529618 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.529544    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.531193 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.531171    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.531272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.531251    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.586473 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.586415    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.586555 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.586497    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwk2\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.651687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.651311    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:06.721272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.721243    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"]
Nov 01 20:41:06.724722 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:06.724687    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eb4e5f_4d3d_4205_b4ca_29a17e6965b7.slice/crio-44d2c47f8897a12abef8643dea75a7996a1028ac199f66c567ad44b0711c1a97 WatchSource:0}: Error finding container 44d2c47f8897a12abef8643dea75a7996a1028ac199f66c567ad44b0711c1a97: Status 404 returned error can't find the container with id 44d2c47f8897a12abef8643dea75a7996a1028ac199f66c567ad44b0711c1a97
Nov 01 20:41:06.727348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.727311    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:06.727412 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.727391    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:06.727453 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.727428    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:06.738437 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.738410    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" event={"ID":"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7","Type":"ContainerStarted","Data":"44d2c47f8897a12abef8643dea75a7996a1028ac199f66c567ad44b0711c1a97"}
Nov 01 20:41:06.740204 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.740179    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" event={"ID":"6d37b7fd-beb4-428c-8f6b-ca28097b15a9","Type":"ContainerStarted","Data":"1701d4ebc897a8b22bf1c64d0a54658f6c1726a27f78c96f9c334f17807b967b"}
Nov 01 20:41:06.740319 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.740226    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" event={"ID":"6d37b7fd-beb4-428c-8f6b-ca28097b15a9","Type":"ContainerStarted","Data":"48099fbffa332b9cd0df625f9b7dc4411b7805fa4181037166727630af2e95e7"}
Nov 01 20:41:06.859924 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:06.859824    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podStartSLOduration=2.85980989 podStartE2EDuration="2.85980989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:06.856222236 +0000 UTC m=+623.905332054" watchObservedRunningTime="2026-11-01 20:41:06.85980989 +0000 UTC m=+623.908919708"
Nov 01 20:41:07.030055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.030026    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"]
Nov 01 20:41:07.031672 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:07.031630    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e9d24b_2ccb_4ade_8a6c_9c7fac6c27e7.slice/crio-2a16a0f1a996ebc443701ba1694742129d75661d1665a7690cc500a8a33703fa WatchSource:0}: Error finding container 2a16a0f1a996ebc443701ba1694742129d75661d1665a7690cc500a8a33703fa: Status 404 returned error can't find the container with id 2a16a0f1a996ebc443701ba1694742129d75661d1665a7690cc500a8a33703fa
Nov 01 20:41:07.034040 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.034004    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:07.034116 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.034086    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:07.034156 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.034127    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:07.326073 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.326045    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:07.330868 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.330844    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:07.746417 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.746379    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" event={"ID":"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7","Type":"ContainerStarted","Data":"20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"}
Nov 01 20:41:07.746600 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.746440    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" event={"ID":"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7","Type":"ContainerStarted","Data":"2a16a0f1a996ebc443701ba1694742129d75661d1665a7690cc500a8a33703fa"}
Nov 01 20:41:07.747764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.747740    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" event={"ID":"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7","Type":"ContainerStarted","Data":"78a64526cae75720136b6463645a78b07864d68b196cace04395b27d077f4ad9"}
Nov 01 20:41:07.747971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.747943    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:07.748899 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.748880    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:07.867753 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.867701    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podStartSLOduration=1.867687814 podStartE2EDuration="1.867687814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:07.863898626 +0000 UTC m=+624.913008444" watchObservedRunningTime="2026-11-01 20:41:07.867687814 +0000 UTC m=+624.916797631"
Nov 01 20:41:07.985978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:07.985919    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podStartSLOduration=2.9859004479999998 podStartE2EDuration="2.985900448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:07.985792197 +0000 UTC m=+625.034902016" watchObservedRunningTime="2026-11-01 20:41:07.985900448 +0000 UTC m=+625.035010266"
Nov 01 20:41:08.333108 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.333077    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:08.337763 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.337737    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:08.651601 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.651565    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:08.656624 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.656601    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:08.753671 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.753628    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:08.753842 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.753713    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:08.754517 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.754493    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:41:08.754628 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:08.754544    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:41:14.801468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.801436    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"]
Nov 01 20:41:14.806108 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.806091    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:14.860433 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.860401    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"]
Nov 01 20:41:14.909464 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.909434    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-cd248f\""
Nov 01 20:41:14.909686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.909560    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-cd248f-dockercfg-xvs4n\""
Nov 01 20:41:14.919886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.919860    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:14.920014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:14.919898    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkx5\" (UniqueName: \"kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5\") pod \"payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.020730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.020701    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.020730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.020736    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkx5\" (UniqueName: \"kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5\") pod \"payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.021646 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.021618    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.078072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.078014    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkx5\" (UniqueName: \"kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5\") pod \"payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.118868 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.118844    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.490374 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.490353    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"]
Nov 01 20:41:15.492384 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:15.492359    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162ab2c2_db87_4fb7_b743_2c3c44b6a795.slice/crio-398c4cdeca795cc3a37d8cde127645119ba77aef164b82da5fe327e265de83be WatchSource:0}: Error finding container 398c4cdeca795cc3a37d8cde127645119ba77aef164b82da5fe327e265de83be: Status 404 returned error can't find the container with id 398c4cdeca795cc3a37d8cde127645119ba77aef164b82da5fe327e265de83be
Nov 01 20:41:15.572918 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.572895    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"]
Nov 01 20:41:15.576541 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.576525    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.624667 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.624621    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.624821 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.624800    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.624889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.624861    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682gj\" (UniqueName: \"kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.624944 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.624924    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.634023 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.633915    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"]
Nov 01 20:41:15.636211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.636193    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-944263-serving-cert\""
Nov 01 20:41:15.726212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.726179    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.726339 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.726252    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682gj\" (UniqueName: \"kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.726339 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.726284    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.726339 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:15.726310    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-serving-cert: secret "maas-api-e2e-worker-w3-944263-serving-cert" not found
Nov 01 20:41:15.726339 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.726318    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.726539 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:15.726371    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls podName:a0b811c2-632f-4dba-ade7-b3e9a8695e92 nodeName:}" failed. No retries permitted until 2026-11-01 20:41:16.226354723 +0000 UTC m=+633.275464533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls") pod "maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" (UID: "a0b811c2-632f-4dba-ade7-b3e9a8695e92") : secret "maas-api-e2e-worker-w3-944263-serving-cert" not found
Nov 01 20:41:15.726953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.726931    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.728872 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.728852    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.784717 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.784666    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-682gj\" (UniqueName: \"kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:15.788126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.788096    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" event={"ID":"162ab2c2-db87-4fb7-b743-2c3c44b6a795","Type":"ContainerStarted","Data":"53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a"}
Nov 01 20:41:15.788221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.788155    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" event={"ID":"162ab2c2-db87-4fb7-b743-2c3c44b6a795","Type":"ContainerStarted","Data":"398c4cdeca795cc3a37d8cde127645119ba77aef164b82da5fe327e265de83be"}
Nov 01 20:41:15.788221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.788185    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:15.902504 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:15.902461    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" podStartSLOduration=1.902449453 podStartE2EDuration="1.902449453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:15.901747796 +0000 UTC m=+632.950857628" watchObservedRunningTime="2026-11-01 20:41:15.902449453 +0000 UTC m=+632.951559271"
Nov 01 20:41:16.231748 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:16.231716    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:16.234118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:16.234087    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:16.488774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:16.488696    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:16.909973 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:16.909948    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"]
Nov 01 20:41:16.912160 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:16.912136    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b811c2_632f_4dba_ade7_b3e9a8695e92.slice/crio-b17c4b06acaf4af11c83ceb9b0c4e6d0f55e95233ac69a9e32b5c66d24ded257 WatchSource:0}: Error finding container b17c4b06acaf4af11c83ceb9b0c4e6d0f55e95233ac69a9e32b5c66d24ded257: Status 404 returned error can't find the container with id b17c4b06acaf4af11c83ceb9b0c4e6d0f55e95233ac69a9e32b5c66d24ded257
Nov 01 20:41:16.914996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:16.914976    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:41:17.798199 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:17.798168    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" event={"ID":"a0b811c2-632f-4dba-ade7-b3e9a8695e92","Type":"ContainerStarted","Data":"b17c4b06acaf4af11c83ceb9b0c4e6d0f55e95233ac69a9e32b5c66d24ded257"}
Nov 01 20:41:18.804453 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:18.804420    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" event={"ID":"a0b811c2-632f-4dba-ade7-b3e9a8695e92","Type":"ContainerStarted","Data":"cc85cfc34dcd92fdfe8a0a31b6506131cd1374abb56d27569fd3c64d4b232636"}
Nov 01 20:41:18.804871 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:18.804550    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:18.923958 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:18.923901    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" podStartSLOduration=3.042550028 podStartE2EDuration="3.923886011s" totalImagesPullingTime="881.335983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:18.921933416 +0000 UTC m=+635.971043236" watchObservedRunningTime="2026-11-01 20:41:18.923886011 +0000 UTC m=+635.972995828"
Nov 01 20:41:20.652940 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.652882    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"]
Nov 01 20:41:20.656908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.656891    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.675604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.675578    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fnr\" (UniqueName: \"kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.675708 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.675620    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.675757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.675741    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.675812 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.675800    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.710931 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.710908    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-dd038b-serving-cert\""
Nov 01 20:41:20.763602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.763578    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"]
Nov 01 20:41:20.776722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.776700    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.776834 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.776751    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.776834 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.776797    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fnr\" (UniqueName: \"kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.776834 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.776830    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.776991 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:20.776838    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-serving-cert: secret "maas-api-e2e-worker-w6-dd038b-serving-cert" not found
Nov 01 20:41:20.776991 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:20.776913    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls podName:e9380fa8-60b4-49f5-9653-a6940faa6b9e nodeName:}" failed. No retries permitted until 2026-11-01 20:41:21.276893972 +0000 UTC m=+638.326003782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls") pod "maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" (UID: "e9380fa8-60b4-49f5-9653-a6940faa6b9e") : secret "maas-api-e2e-worker-w6-dd038b-serving-cert" not found
Nov 01 20:41:20.777482 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.777463    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.778996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.778978    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:20.836211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:20.836180    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fnr\" (UniqueName: \"kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:21.282524 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:21.282486    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:21.284906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:21.284881    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:21.569714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:21.569587    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:21.945247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:21.945226    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"]
Nov 01 20:41:21.946740 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:21.946713    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9380fa8_60b4_49f5_9653_a6940faa6b9e.slice/crio-2ae54938b6a121460b933af0fa5f623088a0827d4cb8b477e7e7b4118f5cf0b5 WatchSource:0}: Error finding container 2ae54938b6a121460b933af0fa5f623088a0827d4cb8b477e7e7b4118f5cf0b5: Status 404 returned error can't find the container with id 2ae54938b6a121460b933af0fa5f623088a0827d4cb8b477e7e7b4118f5cf0b5
Nov 01 20:41:22.827972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:22.827946    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" event={"ID":"e9380fa8-60b4-49f5-9653-a6940faa6b9e","Type":"ContainerStarted","Data":"2ae54938b6a121460b933af0fa5f623088a0827d4cb8b477e7e7b4118f5cf0b5"}
Nov 01 20:41:23.364627 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:23.364593    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:41:23.833533 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:23.833501    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" event={"ID":"e9380fa8-60b4-49f5-9653-a6940faa6b9e","Type":"ContainerStarted","Data":"a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec"}
Nov 01 20:41:23.833729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:23.833612    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:23.952988 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:23.952926    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" podStartSLOduration=3.111721282 podStartE2EDuration="3.95290517s" totalImagesPullingTime="841.183888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:23.951220087 +0000 UTC m=+641.000329906" watchObservedRunningTime="2026-11-01 20:41:23.95290517 +0000 UTC m=+641.002014991"
Nov 01 20:41:24.816730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:24.816692    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 20:41:25.554406 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.554377    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"]
Nov 01 20:41:25.558135 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.558115    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.634162 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.634131    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.634310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.634286    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.634376 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.634325    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9n26\" (UniqueName: \"kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.634376 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.634343    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.664935 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.664913    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-db0157-serving-cert\""
Nov 01 20:41:25.667138 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.667119    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"]
Nov 01 20:41:25.735706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.735677    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.735849 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.735716    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9n26\" (UniqueName: \"kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.735849 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.735737    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.735849 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.735774    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.736012 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:25.735849    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-serving-cert: secret "maas-api-e2e-worker-w1-db0157-serving-cert" not found
Nov 01 20:41:25.736012 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:25.735921    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls podName:fff7ea87-900f-4ece-9d58-d18da67fb3a5 nodeName:}" failed. No retries permitted until 2026-11-01 20:41:26.235902848 +0000 UTC m=+643.285012659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls") pod "maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" (UID: "fff7ea87-900f-4ece-9d58-d18da67fb3a5") : secret "maas-api-e2e-worker-w1-db0157-serving-cert" not found
Nov 01 20:41:25.736366 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.736346    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.738098 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.738075    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:25.793418 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:25.793395    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9n26\" (UniqueName: \"kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:26.241759 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:26.241724    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:26.244099 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:26.244064    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-db0157-69688596fc-l76xd\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:26.470516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:26.470484    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:26.795682 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:26.795608    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 20:41:26.841929 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:26.841905    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"]
Nov 01 20:41:26.843675 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:26.843637    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff7ea87_900f_4ece_9d58_d18da67fb3a5.slice/crio-73b3b63b240990d2b7ecf58d56a0ed0d3f40bdcef5f73463064b770657f7d98d WatchSource:0}: Error finding container 73b3b63b240990d2b7ecf58d56a0ed0d3f40bdcef5f73463064b770657f7d98d: Status 404 returned error can't find the container with id 73b3b63b240990d2b7ecf58d56a0ed0d3f40bdcef5f73463064b770657f7d98d
Nov 01 20:41:27.854056 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:27.854021    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" event={"ID":"fff7ea87-900f-4ece-9d58-d18da67fb3a5","Type":"ContainerStarted","Data":"994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca"}
Nov 01 20:41:27.854400 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:27.854071    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" event={"ID":"fff7ea87-900f-4ece-9d58-d18da67fb3a5","Type":"ContainerStarted","Data":"73b3b63b240990d2b7ecf58d56a0ed0d3f40bdcef5f73463064b770657f7d98d"}
Nov 01 20:41:27.854400 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:27.854144    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:27.970910 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:27.970831    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" podStartSLOduration=2.15551431 podStartE2EDuration="2.97081895s" totalImagesPullingTime="815.30464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:27.969062363 +0000 UTC m=+645.018172193" watchObservedRunningTime="2026-11-01 20:41:27.97081895 +0000 UTC m=+645.019928768"
Nov 01 20:41:29.843080 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:29.843054    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 20:41:33.866648 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:33.866617    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 20:41:35.278698 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.278651    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"]
Nov 01 20:41:35.282745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.282723    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.389068 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.389041    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-208f2b-serving-cert\""
Nov 01 20:41:35.391274 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.391253    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"]
Nov 01 20:41:35.444636 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.444604    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.444793 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.444735    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrwr\" (UniqueName: \"kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.444793 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.444769    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.444908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.444876    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.545940 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.545862    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.545940 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.545917    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrwr\" (UniqueName: \"kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.546127 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.545945    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.546127 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.546017    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.546235 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:35.546191    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-serving-cert: secret "maas-api-e2e-worker-w0-208f2b-serving-cert" not found
Nov 01 20:41:35.546290 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:35.546281    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls podName:4dbcc058-f89d-47f3-b56d-18881e425ca6 nodeName:}" failed. No retries permitted until 2026-11-01 20:41:36.046262907 +0000 UTC m=+653.095372730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls") pod "maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" (UID: "4dbcc058-f89d-47f3-b56d-18881e425ca6") : secret "maas-api-e2e-worker-w0-208f2b-serving-cert" not found
Nov 01 20:41:35.546731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.546709    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.548320 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.548283    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:35.612730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:35.607133    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrwr\" (UniqueName: \"kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:36.050221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:36.050185    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:36.052580 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:36.052550    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:36.208841 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:36.208809    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:36.587797 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:36.587773    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"]
Nov 01 20:41:36.592967 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:36.592934    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbcc058_f89d_47f3_b56d_18881e425ca6.slice/crio-372b31f66e6daacda75537b7a8f2941c2737de090879ded5a766bf0a1b02d7ad WatchSource:0}: Error finding container 372b31f66e6daacda75537b7a8f2941c2737de090879ded5a766bf0a1b02d7ad: Status 404 returned error can't find the container with id 372b31f66e6daacda75537b7a8f2941c2737de090879ded5a766bf0a1b02d7ad
Nov 01 20:41:36.924173 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:36.924134    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" event={"ID":"4dbcc058-f89d-47f3-b56d-18881e425ca6","Type":"ContainerStarted","Data":"372b31f66e6daacda75537b7a8f2941c2737de090879ded5a766bf0a1b02d7ad"}
Nov 01 20:41:37.934236 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:37.934142    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" event={"ID":"4dbcc058-f89d-47f3-b56d-18881e425ca6","Type":"ContainerStarted","Data":"7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b"}
Nov 01 20:41:37.934594 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:37.934301    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:38.050855 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:38.050809    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" podStartSLOduration=2.006923926 podStartE2EDuration="3.050796453s" totalImagesPullingTime="1.043872527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:38.049165861 +0000 UTC m=+655.098275680" watchObservedRunningTime="2026-11-01 20:41:38.050796453 +0000 UTC m=+655.099906269"
Nov 01 20:41:40.148189 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.148155    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"]
Nov 01 20:41:40.152134 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.152116    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.207006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.206966    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-cd248f-serving-cert\""
Nov 01 20:41:40.259637 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.259608    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"]
Nov 01 20:41:40.296878 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.296848    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.297006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.296978    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.297152 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.297125    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.297208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.297191    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2m25\" (UniqueName: \"kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.398642 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.398577    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.398642 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.398617    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2m25\" (UniqueName: \"kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.398827 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.398667    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.398827 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.398704    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.398827 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:40.398817    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-serving-cert: secret "maas-api-e2e-worker-w2-cd248f-serving-cert" not found
Nov 01 20:41:40.399060 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:40.398870    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls podName:f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1 nodeName:}" failed. No retries permitted until 2026-11-01 20:41:40.89885567 +0000 UTC m=+657.947965467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls") pod "maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" (UID: "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1") : secret "maas-api-e2e-worker-w2-cd248f-serving-cert" not found
Nov 01 20:41:40.399561 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.399537    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.401134 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.401107    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.457759 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.457729    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2m25\" (UniqueName: \"kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.904357 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.904319    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:40.906622 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:40.906596    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:41.065334 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.065302    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:41.441604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.441577    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"]
Nov 01 20:41:41.443997 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:41:41.443970    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5baa56e_8648_45ab_9c9f_cc4b1cae7dc1.slice/crio-5bfc52228a998d1fada919e6bef9efb09d9f8b517faf42244959b6603bddf083 WatchSource:0}: Error finding container 5bfc52228a998d1fada919e6bef9efb09d9f8b517faf42244959b6603bddf083: Status 404 returned error can't find the container with id 5bfc52228a998d1fada919e6bef9efb09d9f8b517faf42244959b6603bddf083
Nov 01 20:41:41.958472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.958430    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" event={"ID":"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1","Type":"ContainerStarted","Data":"5bfc52228a998d1fada919e6bef9efb09d9f8b517faf42244959b6603bddf083"}
Nov 01 20:41:41.960467 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.960437    4872 generic.go:356] "Generic (PLEG): container finished" podID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerID="1701d4ebc897a8b22bf1c64d0a54658f6c1726a27f78c96f9c334f17807b967b" exitCode=0
Nov 01 20:41:41.960586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.960509    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" event={"ID":"6d37b7fd-beb4-428c-8f6b-ca28097b15a9","Type":"ContainerDied","Data":"1701d4ebc897a8b22bf1c64d0a54658f6c1726a27f78c96f9c334f17807b967b"}
Nov 01 20:41:41.960984 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.960966    4872 scope.go:122] "RemoveContainer" containerID="1701d4ebc897a8b22bf1c64d0a54658f6c1726a27f78c96f9c334f17807b967b"
Nov 01 20:41:41.962304 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.962274    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:41.962420 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.962346    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:41.962420 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:41.962376    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:41:42.967678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:42.967630    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" event={"ID":"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1","Type":"ContainerStarted","Data":"c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722"}
Nov 01 20:41:42.968140 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:42.967811    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:42.969519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:42.969494    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" event={"ID":"6d37b7fd-beb4-428c-8f6b-ca28097b15a9","Type":"ContainerStarted","Data":"fec81527bba23dae758484a974e222193475e6c6e6d294e05ea7b3a862090c62"}
Nov 01 20:41:43.085676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:43.085616    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" podStartSLOduration=2.274777742 podStartE2EDuration="3.085603582s" totalImagesPullingTime="810.82584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:43.08514183 +0000 UTC m=+660.134251637" watchObservedRunningTime="2026-11-01 20:41:43.085603582 +0000 UTC m=+660.134713399"
Nov 01 20:41:43.325707 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:43.325593    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:43.325883 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:43.325819    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 20:41:43.325883 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:43.325867    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 20:41:43.943986 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:43.943959    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 20:41:44.326145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:44.326062    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 20:41:44.326145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:44.326111    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 20:41:45.326630 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:45.326598    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 20:41:45.327051 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:45.326676    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 20:41:46.326438 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:46.326400    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 20:41:46.326616 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:46.326453    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 20:41:47.326073 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:47.326037    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 20:41:47.326488 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:47.326091    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 20:41:48.330026 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:48.330000    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:48.330451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:48.330262    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:48.330996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:48.330980    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 20:41:48.979719 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:41:48.979691    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 20:41:53.367535 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:41:53.367504    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:42:01.984586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:01.984495    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"]
Nov 01 20:42:01.991145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:01.991105    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.095547 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.095513    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"openshift-service-ca.crt\""
Nov 01 20:42:02.095547 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.095536    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"kube-root-ca.crt\""
Nov 01 20:42:02.095547 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.095553    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"default-dockercfg-9d4fj\""
Nov 01 20:42:02.095839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.095773    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"simulator-w3-944263-kserve-self-signed-certs\""
Nov 01 20:42:02.097991 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.097973    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"]
Nov 01 20:42:02.119935 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.119907    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.120082 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.120032    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.120082 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.120072    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.120253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.120229    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcv8\" (UniqueName: \"kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.120323 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.120307    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.120383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.120353    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.221530 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.221499    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.221771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.221582    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.221771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.221603    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.221771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.221632    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcv8\" (UniqueName: \"kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.221771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.221722    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.221771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.221751    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.222059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.222034    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.222106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.222056    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.222155 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.222120    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.224035 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.224012    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.224167 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.224143    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.279121 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.279041    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcv8\" (UniqueName: \"kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8\") pod \"simulator-w3-944263-kserve-665686fdcd-hcjdz\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") " pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.314613 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.314574    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:02.689753 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:02.689720    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"]
Nov 01 20:42:02.691779 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:42:02.691745    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c93251_4cd5_43b6_af0a_6d7af6995c50.slice/crio-631daafc6adba3447f9a5a0b9328eecf68b2581e42b0036bf8a2deddd5f94ff9 WatchSource:0}: Error finding container 631daafc6adba3447f9a5a0b9328eecf68b2581e42b0036bf8a2deddd5f94ff9: Status 404 returned error can't find the container with id 631daafc6adba3447f9a5a0b9328eecf68b2581e42b0036bf8a2deddd5f94ff9
Nov 01 20:42:03.077897 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:03.077814    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" event={"ID":"37c93251-4cd5-43b6-af0a-6d7af6995c50","Type":"ContainerStarted","Data":"77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2"}
Nov 01 20:42:03.077897 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:03.077841    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" event={"ID":"37c93251-4cd5-43b6-af0a-6d7af6995c50","Type":"ContainerStarted","Data":"631daafc6adba3447f9a5a0b9328eecf68b2581e42b0036bf8a2deddd5f94ff9"}
Nov 01 20:42:03.190551 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:03.190499    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podStartSLOduration=2.190485765 podStartE2EDuration="2.190485765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:03.188740576 +0000 UTC m=+680.237850394" watchObservedRunningTime="2026-11-01 20:42:03.190485765 +0000 UTC m=+680.239595582"
Nov 01 20:42:03.314864 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:03.314833    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:12.318704 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:12.318104    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:12.330203 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:12.330174    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:13.160928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:13.160899    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:42:16.167010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:16.166925    4872 generic.go:356] "Generic (PLEG): container finished" podID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerID="20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee" exitCode=0
Nov 01 20:42:16.167010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:16.166992    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" event={"ID":"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7","Type":"ContainerDied","Data":"20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"}
Nov 01 20:42:16.167474 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:16.167396    4872 scope.go:122] "RemoveContainer" containerID="20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"
Nov 01 20:42:16.168651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:16.168623    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:42:16.168738 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:16.168705    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:42:16.168778 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:16.168741    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:42:17.181704 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:17.179955    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" event={"ID":"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7","Type":"ContainerStarted","Data":"28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae"}
Nov 01 20:42:17.652140 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:17.652103    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:42:17.652348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:17.652233    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:42:17.652348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:17.652284    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 20:42:18.652173 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:18.652137    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:42:18.652646 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:18.652201    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 20:42:19.651933 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:19.651892    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:42:19.652123 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:19.651949    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 20:42:20.652061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:20.652022    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:42:20.652464 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:20.652092    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 20:42:21.652294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:21.652251    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:42:21.652716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:21.652315    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 20:42:22.656671 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:22.656621    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:42:22.657171 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:22.656942    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:42:22.657547 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:22.657526    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 20:42:23.371075 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:42:23.371033    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:42:23.647074 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.647042    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"]
Nov 01 20:42:23.652029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.652004    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.714593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.714558    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"]
Nov 01 20:42:23.727787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.727747    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.728030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.727853    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthrw\" (UniqueName: \"kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.728030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.727909    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.728030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.727938    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.728166 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.728104    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.728213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.728164    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.760674 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.760622    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"default-dockercfg-hdkp4\""
Nov 01 20:42:23.760674 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.760622    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"openshift-service-ca.crt\""
Nov 01 20:42:23.760905 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.760701    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"simulator-w6-dd038b-kserve-self-signed-certs\""
Nov 01 20:42:23.760905 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.760761    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"kube-root-ca.crt\""
Nov 01 20:42:23.828961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.828923    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.828979    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthrw\" (UniqueName: \"kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829019    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829053    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829106    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829132    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829434 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829392    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829700 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829646    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.829783 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.829711    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.831523 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.831493    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.831523 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.831520    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.886859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.886822    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthrw\" (UniqueName: \"kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw\") pod \"simulator-w6-dd038b-kserve-6f89d55d48-t755n\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") " pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:23.965760 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:23.965664    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:24.348614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:24.348585    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"]
Nov 01 20:42:24.350568 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:42:24.350540    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb47e85a_b794_4b0e_b415_83a7191b4672.slice/crio-5c7f5385203c70b34994daf6739adebe2888e082420509f35cacf85b0fb392d0 WatchSource:0}: Error finding container 5c7f5385203c70b34994daf6739adebe2888e082420509f35cacf85b0fb392d0: Status 404 returned error can't find the container with id 5c7f5385203c70b34994daf6739adebe2888e082420509f35cacf85b0fb392d0
Nov 01 20:42:24.795807 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:24.795773    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:42:24.796318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:24.795827    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:42:25.272034 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:25.272000    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" event={"ID":"db47e85a-b794-4b0e-b415-83a7191b4672","Type":"ContainerStarted","Data":"e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c"}
Nov 01 20:42:25.272034 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:25.272040    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" event={"ID":"db47e85a-b794-4b0e-b415-83a7191b4672","Type":"ContainerStarted","Data":"5c7f5385203c70b34994daf6739adebe2888e082420509f35cacf85b0fb392d0"}
Nov 01 20:42:25.389261 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:25.389209    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podStartSLOduration=2.389194312 podStartE2EDuration="2.389194312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:25.387207577 +0000 UTC m=+702.436317396" watchObservedRunningTime="2026-11-01 20:42:25.389194312 +0000 UTC m=+702.438304133"
Nov 01 20:42:25.876629 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:42:25.876589    4872 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eb4e5f_4d3d_4205_b4ca_29a17e6965b7.slice/crio-conmon-78a64526cae75720136b6463645a78b07864d68b196cace04395b27d077f4ad9.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 20:42:25.966112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:25.966070    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:26.279501 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:26.279468    4872 generic.go:356] "Generic (PLEG): container finished" podID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerID="78a64526cae75720136b6463645a78b07864d68b196cace04395b27d077f4ad9" exitCode=0
Nov 01 20:42:26.279732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:26.279541    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" event={"ID":"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7","Type":"ContainerDied","Data":"78a64526cae75720136b6463645a78b07864d68b196cace04395b27d077f4ad9"}
Nov 01 20:42:26.279996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:26.279980    4872 scope.go:122] "RemoveContainer" containerID="78a64526cae75720136b6463645a78b07864d68b196cace04395b27d077f4ad9"
Nov 01 20:42:26.281254 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:26.281227    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:42:26.281333 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:26.281299    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:42:26.281373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:26.281334    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:42:27.286481 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:27.286442    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" event={"ID":"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7","Type":"ContainerStarted","Data":"9c24561ef41ee91bb4a6a2f75bde907e99a399ddef66a1825b02eb6e4e478879"}
Nov 01 20:42:27.333231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:27.333192    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:42:27.333418 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:27.333394    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 20:42:27.333466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:27.333445    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 20:42:28.333741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.333706    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 20:42:28.334235 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.333767    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 20:42:28.818778 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.818729    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"]
Nov 01 20:42:28.822635 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.822603    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.858343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.858308    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"]
Nov 01 20:42:28.889612 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.889578    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8czr\" (UniqueName: \"kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.889802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.889637    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.889802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.889752    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.889879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.889808    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.889920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.889900    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.889958 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.889935    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.918713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.918014    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"default-dockercfg-dssl5\""
Nov 01 20:42:28.918713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.918111    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"simulator-w1-db0157-kserve-self-signed-certs\""
Nov 01 20:42:28.926561 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.926525    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"openshift-service-ca.crt\""
Nov 01 20:42:28.926561 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.926534    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"kube-root-ca.crt\""
Nov 01 20:42:28.990584 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.990535    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.990824 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.990623    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8czr\" (UniqueName: \"kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.990824 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.990669    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.990824 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.990700    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.990824 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.990738    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.990824 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.990793    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.991092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.991069    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.991092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.991083    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.991161 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.991141    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.993225 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.993199    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:28.993385 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:28.993362    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:29.048437 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:29.048399    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8czr\" (UniqueName: \"kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr\") pod \"simulator-w1-db0157-kserve-7874796795-gdjjh\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") " pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:29.136162 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:29.136076    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:29.333474 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:29.333440    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 20:42:29.333675 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:29.333491    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 20:42:29.521681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:29.521639    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"]
Nov 01 20:42:29.522904 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:42:29.522872    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22847bb6_c2c4_475e_bfc5_f1d76ffa72f7.slice/crio-b666c308a32354edb9ce7d229e3ade8ae7d349ab95863a6ea690abd9779703c3 WatchSource:0}: Error finding container b666c308a32354edb9ce7d229e3ade8ae7d349ab95863a6ea690abd9779703c3: Status 404 returned error can't find the container with id b666c308a32354edb9ce7d229e3ade8ae7d349ab95863a6ea690abd9779703c3
Nov 01 20:42:30.332748 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:30.332715    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 20:42:30.332936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:30.332765    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 20:42:30.333165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:30.333142    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" event={"ID":"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7","Type":"ContainerStarted","Data":"0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02"}
Nov 01 20:42:30.333253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:30.333183    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" event={"ID":"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7","Type":"ContainerStarted","Data":"b666c308a32354edb9ce7d229e3ade8ae7d349ab95863a6ea690abd9779703c3"}
Nov 01 20:42:30.450387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:30.450242    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podStartSLOduration=2.450222685 podStartE2EDuration="2.450222685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:30.448245893 +0000 UTC m=+707.497355711" watchObservedRunningTime="2026-11-01 20:42:30.450222685 +0000 UTC m=+707.499332504"
Nov 01 20:42:31.136935 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:31.136899    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:31.333671 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:31.333619    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 20:42:31.333849 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:31.333716    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 20:42:32.346974 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:32.346934    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:42:32.347925 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:32.347905    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:42:32.348543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:32.348528    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 20:42:33.966699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:33.966668    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:33.979134 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:33.979108    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:34.365649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:34.365568    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:42:39.136925 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:39.136893    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:39.149411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:39.149386    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:39.394342 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:39.394259    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:42:53.369628 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:42:53.369588    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:42:58.650328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.650289    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"]
Nov 01 20:42:58.654112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.654093    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.727741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.727708    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.727741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.727744    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.727928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.727836    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.727928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.727886    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.728001 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.727946    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.728121 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.728095    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnxh\" (UniqueName: \"kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.757948 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.757923    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"premium-w3-944263-kserve-self-signed-certs\""
Nov 01 20:42:58.760080 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.760058    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"]
Nov 01 20:42:58.829052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829019    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829052 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829058    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829290 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829113    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829290 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829141    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829290 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829198    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829290 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829237    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnxh\" (UniqueName: \"kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829537 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829513    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829569    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.829587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.829582    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.831437 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.831415    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.831567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.831552    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.886836 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.886805    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnxh\" (UniqueName: \"kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh\") pod \"premium-w3-944263-kserve-7f5b4bff8c-bzklz\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") " pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:58.965947 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:58.965856    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:42:59.343848 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:59.343821    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"]
Nov 01 20:42:59.345471 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:42:59.345431    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod621e908b_85a1_48af_916d_aaafcb7f15b6.slice/crio-1e0f86cef5da26f3322ae4b15c91520f914d1ebf092e8dbbae7a4ab4308d7400 WatchSource:0}: Error finding container 1e0f86cef5da26f3322ae4b15c91520f914d1ebf092e8dbbae7a4ab4308d7400: Status 404 returned error can't find the container with id 1e0f86cef5da26f3322ae4b15c91520f914d1ebf092e8dbbae7a4ab4308d7400
Nov 01 20:42:59.488055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:59.488021    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" event={"ID":"621e908b-85a1-48af-916d-aaafcb7f15b6","Type":"ContainerStarted","Data":"144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce"}
Nov 01 20:42:59.488055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:59.488069    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" event={"ID":"621e908b-85a1-48af-916d-aaafcb7f15b6","Type":"ContainerStarted","Data":"1e0f86cef5da26f3322ae4b15c91520f914d1ebf092e8dbbae7a4ab4308d7400"}
Nov 01 20:42:59.623264 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:59.623161    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podStartSLOduration=1.623148016 podStartE2EDuration="1.623148016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:59.620524997 +0000 UTC m=+736.669634827" watchObservedRunningTime="2026-11-01 20:42:59.623148016 +0000 UTC m=+736.672257834"
Nov 01 20:42:59.966769 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:42:59.966731    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:43:08.966388 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:08.966354    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:43:08.978811 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:08.978788    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:43:09.549923 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:09.549897    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:43:10.302408 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.302359    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"]
Nov 01 20:43:10.306540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.306521    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.354285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.354257    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.354473 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.354321    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.354473 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.354454    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.354583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.354496    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.354583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.354526    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvrl\" (UniqueName: \"kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.354674 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.354612    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.362828 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.362796    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"]
Nov 01 20:43:10.410485 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.410454    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"kube-root-ca.crt\""
Nov 01 20:43:10.410485 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.410461    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"openshift-service-ca.crt\""
Nov 01 20:43:10.410713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.410570    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"simulator-w4-1a1e9e-kserve-self-signed-certs\""
Nov 01 20:43:10.410713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.410593    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"default-dockercfg-rp528\""
Nov 01 20:43:10.456069 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456011    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.456321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456087    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.456321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456126    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.456321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456153    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.456321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456177    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvrl\" (UniqueName: \"kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.456321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456245    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.456906 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.456880    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.457154 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.457126    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.458136 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.458105    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.461677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.460364    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.461677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.460427    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.514760 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.514727    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvrl\" (UniqueName: \"kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl\") pod \"simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") " pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.619318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.619222    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:10.995257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:10.995228    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"]
Nov 01 20:43:10.996860 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:43:10.996831    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11224b1_812a_4361_8292_f4d8e2b44095.slice/crio-5106f3295218030aa452eb1e6802c8db08b122c92815ad111438fc9c32239c27 WatchSource:0}: Error finding container 5106f3295218030aa452eb1e6802c8db08b122c92815ad111438fc9c32239c27: Status 404 returned error can't find the container with id 5106f3295218030aa452eb1e6802c8db08b122c92815ad111438fc9c32239c27
Nov 01 20:43:11.552182 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:11.552147    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" event={"ID":"a11224b1-812a-4361-8292-f4d8e2b44095","Type":"ContainerStarted","Data":"4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2"}
Nov 01 20:43:11.552182 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:11.552187    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" event={"ID":"a11224b1-812a-4361-8292-f4d8e2b44095","Type":"ContainerStarted","Data":"5106f3295218030aa452eb1e6802c8db08b122c92815ad111438fc9c32239c27"}
Nov 01 20:43:11.619638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:11.619605    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:11.667794 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:11.667745    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podStartSLOduration=1.667732381 podStartE2EDuration="1.667732381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:11.666568249 +0000 UTC m=+748.715678066" watchObservedRunningTime="2026-11-01 20:43:11.667732381 +0000 UTC m=+748.716842198"
Nov 01 20:43:20.620209 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:20.620176    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:20.633047 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:20.633021    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:21.620212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:21.620180    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:43:23.368847 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:43:23.368810    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:43:35.747083 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.747008    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"]
Nov 01 20:43:35.751107 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.751083    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.810411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.810372    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"]
Nov 01 20:43:35.842631 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.842601    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.842631 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.842635    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.842859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.842799    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hcs\" (UniqueName: \"kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.842859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.842824    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.842933 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.842867    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.843026 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.843000    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.854630 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.854604    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"premium-w6-dd038b-kserve-self-signed-certs\""
Nov 01 20:43:35.944327 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944298    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944327 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944332    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944409    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hcs\" (UniqueName: \"kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944428    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944473    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944517    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944807 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944781    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944872 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944818    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.944929 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.944893    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.946749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.946725    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:35.946929 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:35.946912    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:36.001806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:36.001757    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hcs\" (UniqueName: \"kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs\") pod \"premium-w6-dd038b-kserve-59547547b5-8xpph\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") " pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:36.064891 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:36.064856    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:36.435605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:36.435576    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"]
Nov 01 20:43:36.437506 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:43:36.437478    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf611828c_fb69_406f_a4a0_6d1c85c49183.slice/crio-0ead97893e60b106a4dfa2f346495c1df837db581ea488e8dac8f508f225820f WatchSource:0}: Error finding container 0ead97893e60b106a4dfa2f346495c1df837db581ea488e8dac8f508f225820f: Status 404 returned error can't find the container with id 0ead97893e60b106a4dfa2f346495c1df837db581ea488e8dac8f508f225820f
Nov 01 20:43:36.694216 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:36.694181    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" event={"ID":"f611828c-fb69-406f-a4a0-6d1c85c49183","Type":"ContainerStarted","Data":"646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb"}
Nov 01 20:43:36.694216 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:36.694222    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" event={"ID":"f611828c-fb69-406f-a4a0-6d1c85c49183","Type":"ContainerStarted","Data":"0ead97893e60b106a4dfa2f346495c1df837db581ea488e8dac8f508f225820f"}
Nov 01 20:43:36.811536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:36.811479    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podStartSLOduration=1.811465316 podStartE2EDuration="1.811465316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:36.809923979 +0000 UTC m=+773.859033797" watchObservedRunningTime="2026-11-01 20:43:36.811465316 +0000 UTC m=+773.860575133"
Nov 01 20:43:37.066030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:37.065961    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:46.065110 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:46.065073    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:46.077328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:46.077300    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:46.757040 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:43:46.757013    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:43:53.364782 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:43:53.364732    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:44:02.096727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.096699    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"]
Nov 01 20:44:02.100438 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.100423    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.202914 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.202891    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"openshift-service-ca.crt\""
Nov 01 20:44:02.205934 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.205894    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"kube-root-ca.crt\""
Nov 01 20:44:02.206087 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.205961    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"default-dockercfg-lksmz\""
Nov 01 20:44:02.206087 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.205967    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"premium-w0-208f2b-kserve-self-signed-certs\""
Nov 01 20:44:02.209580 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.209557    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"]
Nov 01 20:44:02.229960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.229940    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.230059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.229977    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.230059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.230019    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.230357 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.230335    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mk6\" (UniqueName: \"kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.230449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.230375    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.230529 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.230513    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331443 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331409    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mk6\" (UniqueName: \"kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331443 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331445    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331482    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331531    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331551    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331677    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331941    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.331969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.331969    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.332122 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.332001    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.333875 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.333855    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.334141 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.334125    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.391808 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.391750    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mk6\" (UniqueName: \"kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6\") pod \"premium-w0-208f2b-kserve-79b4c4d994-ftfxx\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") " pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.411892 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.411866    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:02.791237 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.791213    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"]
Nov 01 20:44:02.792936 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:44:02.792905    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc01fd7_8b47_435a_93aa_91301624879b.slice/crio-96891ceeeaa8c50a7f5acf549d6f2ea0252eb3070f2cdeb0b9014c365c10ca3b WatchSource:0}: Error finding container 96891ceeeaa8c50a7f5acf549d6f2ea0252eb3070f2cdeb0b9014c365c10ca3b: Status 404 returned error can't find the container with id 96891ceeeaa8c50a7f5acf549d6f2ea0252eb3070f2cdeb0b9014c365c10ca3b
Nov 01 20:44:02.831857 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:02.831834    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" event={"ID":"bbc01fd7-8b47-435a-93aa-91301624879b","Type":"ContainerStarted","Data":"96891ceeeaa8c50a7f5acf549d6f2ea0252eb3070f2cdeb0b9014c365c10ca3b"}
Nov 01 20:44:03.838177 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:03.838139    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" event={"ID":"bbc01fd7-8b47-435a-93aa-91301624879b","Type":"ContainerStarted","Data":"5056f5e519d4af34b9253656d87b518c7390fa85dec29070cb013174d2e0d4bd"}
Nov 01 20:44:03.955978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:03.955935    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podStartSLOduration=1.955921135 podStartE2EDuration="1.955921135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:03.953710497 +0000 UTC m=+801.002820313" watchObservedRunningTime="2026-11-01 20:44:03.955921135 +0000 UTC m=+801.005030952"
Nov 01 20:44:04.412720 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:04.412690    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:12.412189 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:12.412159    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:12.424649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:12.424624    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:12.897411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:12.897386    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:44:23.365240 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:44:23.365194    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:44:51.682781 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.682751    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"]
Nov 01 20:44:51.686708 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.686691    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.752190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.752161    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.752366 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.752215    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nq8\" (UniqueName: \"kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.752444 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.752363    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.752444 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.752404    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.752557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.752538    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.752613 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.752582    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.790803 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.790778    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"default-dockercfg-rfgc8\""
Nov 01 20:44:51.790803 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.790804    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"premium-w5-c9ae21-kserve-self-signed-certs\""
Nov 01 20:44:51.791000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.790823    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"openshift-service-ca.crt\""
Nov 01 20:44:51.791000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.790927    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"kube-root-ca.crt\""
Nov 01 20:44:51.793332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.793312    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"]
Nov 01 20:44:51.854370 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854336    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nq8\" (UniqueName: \"kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854425    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854451    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854523    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854753 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854566    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854753 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854644    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854905 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854886    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.854996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.854975    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.855054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.855037    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.856736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.856715    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.857014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.856995    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:51.911313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:51.911285    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nq8\" (UniqueName: \"kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8\") pod \"premium-w5-c9ae21-kserve-c5857479b-x6smn\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") " pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:52.020796 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:52.020717    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:44:52.400578 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:52.400555    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"]
Nov 01 20:44:52.402505 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:44:52.402477    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25b3d006_8a3e_4506_8d4d_b652f1cf50fa.slice/crio-9e4b302b0a1cf7545f411299399e53ca2df7de59db3e1f235a97c7df7c2b0022 WatchSource:0}: Error finding container 9e4b302b0a1cf7545f411299399e53ca2df7de59db3e1f235a97c7df7c2b0022: Status 404 returned error can't find the container with id 9e4b302b0a1cf7545f411299399e53ca2df7de59db3e1f235a97c7df7c2b0022
Nov 01 20:44:53.107715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:53.107680    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" event={"ID":"25b3d006-8a3e-4506-8d4d-b652f1cf50fa","Type":"ContainerStarted","Data":"8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc"}
Nov 01 20:44:53.107715 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:53.107719    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" event={"ID":"25b3d006-8a3e-4506-8d4d-b652f1cf50fa","Type":"ContainerStarted","Data":"9e4b302b0a1cf7545f411299399e53ca2df7de59db3e1f235a97c7df7c2b0022"}
Nov 01 20:44:53.224062 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:53.224005    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podStartSLOduration=2.223992406 podStartE2EDuration="2.223992406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:53.222679878 +0000 UTC m=+850.271789687" watchObservedRunningTime="2026-11-01 20:44:53.223992406 +0000 UTC m=+850.273102270"
Nov 01 20:44:53.381384 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:44:53.381275    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:44:54.021286 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:44:54.021257    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:45:02.021540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:02.021463    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:45:02.032963 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:02.032941    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:45:02.175213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:02.175188    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:45:07.278627 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.278591    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"]
Nov 01 20:45:07.282788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.282769    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.388280 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.388248    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-db0157-openshift-default-dockercfg-gx5m8\""
Nov 01 20:45:07.390766 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.390747    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"]
Nov 01 20:45:07.441861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.441826    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.441861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.441865    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442107 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.441896    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442107 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.441953    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.442142    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.442177    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442333 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.442309    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7lx\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442399 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.442382    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.442473 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.442451    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544097 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.543999    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544097 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544041    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544173    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544212    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544298    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544325    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544408    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7lx\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544572 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544509    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544572 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544470    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544572 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544565    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544767 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544585    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.544767 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.544681    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.545116 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.545089    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.545532 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.545508    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.546738 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.546716    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.546927 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.546908    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.602200 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.602170    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.602375 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.602295    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7lx\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:07.897034 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:07.897001    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:08.328378 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:08.328353    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"]
Nov 01 20:45:08.329751 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:45:08.329727    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b2a14b_93c1_42c7_bca9_72feabde7ffd.slice/crio-167c215960911896b20681dc0225de722dd9206491180cf2b8962572789a1b2e WatchSource:0}: Error finding container 167c215960911896b20681dc0225de722dd9206491180cf2b8962572789a1b2e: Status 404 returned error can't find the container with id 167c215960911896b20681dc0225de722dd9206491180cf2b8962572789a1b2e
Nov 01 20:45:08.332227 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:08.332190    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:08.332291 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:08.332281    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:08.332329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:08.332313    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:09.226168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:09.226134    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" event={"ID":"c0b2a14b-93c1-42c7-bca9-72feabde7ffd","Type":"ContainerStarted","Data":"6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a"}
Nov 01 20:45:09.226372 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:09.226188    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" event={"ID":"c0b2a14b-93c1-42c7-bca9-72feabde7ffd","Type":"ContainerStarted","Data":"167c215960911896b20681dc0225de722dd9206491180cf2b8962572789a1b2e"}
Nov 01 20:45:09.446573 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:09.446520    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podStartSLOduration=2.446505605 podStartE2EDuration="2.446505605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:09.41984541 +0000 UTC m=+866.468955228" watchObservedRunningTime="2026-11-01 20:45:09.446505605 +0000 UTC m=+866.495615481"
Nov 01 20:45:09.902680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:09.901983    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:09.919699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:09.914782    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:45:09.919699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:09.914842    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:45:10.918828 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:10.918786    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:45:10.919303 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:10.918852    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:45:11.897995 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:11.897957    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:45:11.898282 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:11.898251    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:45:12.911087 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:12.911053    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:45:12.911691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:12.911638    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:45:13.911681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:13.900044    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:45:13.911681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:13.900114    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:45:14.902138 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:14.902103    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:14.902419 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:14.902399    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:14.903062 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:14.903042    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 20:45:23.369015 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:45:23.368974    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:45:36.990102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:36.990064    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"]
Nov 01 20:45:36.990498 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:36.990296    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy" containerID="cri-o://2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda" gracePeriod=30
Nov 01 20:45:37.042390 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:37.042353    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"]
Nov 01 20:45:37.042638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:37.042612    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy" containerID="cri-o://48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396" gracePeriod=30
Nov 01 20:45:42.319251 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.319215    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:45:42.353247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.353222    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:45:42.421184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.421081    4872 generic.go:356] "Generic (PLEG): container finished" podID="1d493d13-62e2-4a95-bab3-d51702727c32" containerID="48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396" exitCode=0
Nov 01 20:45:42.421650 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.421622    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" event={"ID":"1d493d13-62e2-4a95-bab3-d51702727c32","Type":"ContainerDied","Data":"48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396"}
Nov 01 20:45:42.421859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.421715    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj" event={"ID":"1d493d13-62e2-4a95-bab3-d51702727c32","Type":"ContainerDied","Data":"68cf9aa8663bee8f2a3f29266036f0fe7906921b4b0351556aa40ece0cce3dc4"}
Nov 01 20:45:42.421859 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.421747    4872 scope.go:122] "RemoveContainer" containerID="48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396"
Nov 01 20:45:42.421958 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.421887    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"
Nov 01 20:45:42.424458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.424437    4872 generic.go:356] "Generic (PLEG): container finished" podID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerID="2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda" exitCode=0
Nov 01 20:45:42.424574 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.424476    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" event={"ID":"47fa36eb-539d-498a-8bb6-31f6982cc240","Type":"ContainerDied","Data":"2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda"}
Nov 01 20:45:42.424574 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.424486    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"
Nov 01 20:45:42.424574 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.424515    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl" event={"ID":"47fa36eb-539d-498a-8bb6-31f6982cc240","Type":"ContainerDied","Data":"86cab3e00341b5bf415a64dc5337003a299f46f64628b881c0d5dd4244c97391"}
Nov 01 20:45:42.434815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.434788    4872 scope.go:122] "RemoveContainer" containerID="48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396"
Nov 01 20:45:42.435222 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:45:42.435191    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396\": container with ID starting with 48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396 not found: ID does not exist" containerID="48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396"
Nov 01 20:45:42.435318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.435233    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396"} err="failed to get container status \"48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396\": rpc error: code = NotFound desc = could not find container \"48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396\": container with ID starting with 48f0a06ee848b0af2390f5b230bad0effe73d3e391054d9699257c4898849396 not found: ID does not exist"
Nov 01 20:45:42.435318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.435258    4872 scope.go:122] "RemoveContainer" containerID="2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda"
Nov 01 20:45:42.446466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.446447    4872 scope.go:122] "RemoveContainer" containerID="2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda"
Nov 01 20:45:42.446785 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:45:42.446766    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda\": container with ID starting with 2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda not found: ID does not exist" containerID="2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda"
Nov 01 20:45:42.446862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.446795    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda"} err="failed to get container status \"2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda\": rpc error: code = NotFound desc = could not find container \"2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda\": container with ID starting with 2d01dfcec19ae79730b731ff3622824838984c7e14ceb7960e8eeb36c1838dda not found: ID does not exist"
Nov 01 20:45:42.488991 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.488943    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489177 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489024    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489258 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489172    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489263    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489310    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489341    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489394    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489525 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489434    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489525 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489491    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489635 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489560    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489635 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489588    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489633    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489676    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489710    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489744    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.489962 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489782    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489962 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489828    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket\") pod \"47fa36eb-539d-498a-8bb6-31f6982cc240\" (UID: \"47fa36eb-539d-498a-8bb6-31f6982cc240\") "
Nov 01 20:45:42.489962 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.489863    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy\") pod \"1d493d13-62e2-4a95-bab3-d51702727c32\" (UID: \"1d493d13-62e2-4a95-bab3-d51702727c32\") "
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490205    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490602    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490826    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490832    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490866    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490889    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.490891    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa36eb-539d-498a-8bb6-31f6982cc240-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.491014    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.491344    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.491670 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.491634    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.492255 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.491914    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:42.492316 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.492252    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.493502 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.493476    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.493610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.493598    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:42.496818 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.496786    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:42.496818 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.496790    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:42.496818 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.496804    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "kube-api-access-d95l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:42.496818 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.496816    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:45:42.496996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.496806    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo" pod "1d493d13-62e2-4a95-bab3-d51702727c32" (UID: "1d493d13-62e2-4a95-bab3-d51702727c32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:45:42.496996 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.496879    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc" pod "47fa36eb-539d-498a-8bb6-31f6982cc240" (UID: "47fa36eb-539d-498a-8bb6-31f6982cc240"). InnerVolumeSpecName "kube-api-access-nnlrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591615    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d493d13-62e2-4a95-bab3-d51702727c32-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591646    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591678    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591688    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591697    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591706    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d493d13-62e2-4a95-bab3-d51702727c32-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591716    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d95l8\" (UniqueName: \"kubernetes.io/projected/1d493d13-62e2-4a95-bab3-d51702727c32-kube-api-access-d95l8\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.591722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591727    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591735    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591743    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591751    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlrc\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-kube-api-access-nnlrc\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591759    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591767    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591775    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa36eb-539d-498a-8bb6-31f6982cc240-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591783    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d493d13-62e2-4a95-bab3-d51702727c32-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.592070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.591791    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa36eb-539d-498a-8bb6-31f6982cc240-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:42.847336 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.847265    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"]
Nov 01 20:45:42.853846 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.853815    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2zkzj"]
Nov 01 20:45:42.963541 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.963515    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"]
Nov 01 20:45:42.966633 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:42.966607    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8jpjl"]
Nov 01 20:45:43.082112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:43.082082    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" path="/var/lib/kubelet/pods/1d493d13-62e2-4a95-bab3-d51702727c32/volumes"
Nov 01 20:45:43.082687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:43.082650    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" path="/var/lib/kubelet/pods/47fa36eb-539d-498a-8bb6-31f6982cc240/volumes"
Nov 01 20:45:43.178140 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:43.178114    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:45:43.179114 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:43.179094    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:45:43.184729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:43.184710    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:45:43.185512 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:43.185491    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:45:51.672452 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.672416    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"]
Nov 01 20:45:51.672913 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.672900    4872 cpu_manager.go:414] "containerMap: removing container" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy"
Nov 01 20:45:51.672978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.672914    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy"
Nov 01 20:45:51.672978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.672954    4872 cpu_manager.go:414] "containerMap: removing container" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy"
Nov 01 20:45:51.672978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.672959    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy"
Nov 01 20:45:51.673078 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.673032    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fa36eb-539d-498a-8bb6-31f6982cc240" containerName="istio-proxy"
Nov 01 20:45:51.673078 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.673041    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d493d13-62e2-4a95-bab3-d51702727c32" containerName="istio-proxy"
Nov 01 20:45:51.677984 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.677965    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.742804 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.742776    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-944263-openshift-default-dockercfg-s5hxs\""
Nov 01 20:45:51.784297 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.784259    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"]
Nov 01 20:45:51.797321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797288    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797469 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797332    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797469 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797410    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797469 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797459    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797536    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797558    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltw8\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797713    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797809 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797797    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.797875 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.797863    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898429 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898400    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898618 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898454    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898618 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898472    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltw8\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898618 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898515    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898618 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898570    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898631    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898678    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898705    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898736    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.898863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898813    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.899046 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.898920    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.899046 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.899032    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.899124 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.899059    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.899326 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.899308    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.901050 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.901023    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.901306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.901288    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.958938 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.958880    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltw8\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.958938 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.958914    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.986766 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.986742    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"]
Nov 01 20:45:51.991427 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.991407    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:51.991872 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:51.991858    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.048853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.048495    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cd248f-openshift-default-dockercfg-6v4rd\""
Nov 01 20:45:52.052351 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.052322    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"]
Nov 01 20:45:52.100703 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.100671    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.100873 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.100721    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.100873 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.100768    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.100990 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.100865    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcqg\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.100990 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.100962    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.100990 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.100985    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.101074 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.101035    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.101237 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.101213    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.101288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.101273    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.202593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202561    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.202776 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202607    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.202776 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202733    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.202916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202794    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.202916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202857    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.202916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202881    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcqg\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203051 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202937    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203051 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.202976    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203051 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.203019    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203195 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.203074    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203195 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.203125    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203443 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.203420    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.203687    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.203779 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.203698    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.205086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.205066    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.205397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.205379    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.259856 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.259793    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.260077 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.260058    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcqg\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.318120 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.318092    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:52.421461 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.421426    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"]
Nov 01 20:45:52.424017 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:45:52.423471    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854f9bb0_0165_4bcd_8e45_fb91cf705f34.slice/crio-0c94e2c2d10c6a32850840f08d870f19aa62d0b33b412e8179263cda3b89bdd4 WatchSource:0}: Error finding container 0c94e2c2d10c6a32850840f08d870f19aa62d0b33b412e8179263cda3b89bdd4: Status 404 returned error can't find the container with id 0c94e2c2d10c6a32850840f08d870f19aa62d0b33b412e8179263cda3b89bdd4
Nov 01 20:45:52.426066 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.426026    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:52.426142 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.426116    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:52.426177 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.426161    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:52.489454 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.489427    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" event={"ID":"854f9bb0-0165-4bcd-8e45-fb91cf705f34","Type":"ContainerStarted","Data":"8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466"}
Nov 01 20:45:52.489574 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.489476    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" event={"ID":"854f9bb0-0165-4bcd-8e45-fb91cf705f34","Type":"ContainerStarted","Data":"0c94e2c2d10c6a32850840f08d870f19aa62d0b33b412e8179263cda3b89bdd4"}
Nov 01 20:45:52.608533 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.608437    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podStartSLOduration=1.6084188849999999 podStartE2EDuration="1.608418885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:52.606060677 +0000 UTC m=+909.655170494" watchObservedRunningTime="2026-11-01 20:45:52.608418885 +0000 UTC m=+909.657528704"
Nov 01 20:45:52.698244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.698207    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"]
Nov 01 20:45:52.700215 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:45:52.700189    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c263b9_107a_4225_9fa9_21a960aa8c20.slice/crio-f9417648f553bdc841ea610dba88347cdf2d186932096f4bd3c6518f6d0c508f WatchSource:0}: Error finding container f9417648f553bdc841ea610dba88347cdf2d186932096f4bd3c6518f6d0c508f: Status 404 returned error can't find the container with id f9417648f553bdc841ea610dba88347cdf2d186932096f4bd3c6518f6d0c508f
Nov 01 20:45:52.702778 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.702743    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:52.702879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.702811    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:52.702879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.702846    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:52.944269 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.944224    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"]
Nov 01 20:45:52.953419 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.953390    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:52.992430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:52.992402    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033636    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdwd\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033733    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033788    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033835    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033909    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033933    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.033996    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.034017    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.034691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.034084    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.036122 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.035877    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-208f2b-openshift-default-dockercfg-x8lc4\""
Nov 01 20:45:53.037606 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.037517    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"]
Nov 01 20:45:53.135540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135497    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135545    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135586    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135602    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135644    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135722    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdwd\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135749    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.135787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135780    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.136115 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135808    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.136115 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.135986    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.136115 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.136015    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.136277 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.136224    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.136472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.136449    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.136541 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.136520    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.138211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.138187    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.138415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.138393    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.201739 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.199339    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdwd\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.226839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.226763    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.273123 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.273089    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:53.391289 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:45:53.391249    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:45:53.551612 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.551502    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" event={"ID":"c0c263b9-107a-4225-9fa9-21a960aa8c20","Type":"ContainerStarted","Data":"b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c"}
Nov 01 20:45:53.551612 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.551574    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" event={"ID":"c0c263b9-107a-4225-9fa9-21a960aa8c20","Type":"ContainerStarted","Data":"f9417648f553bdc841ea610dba88347cdf2d186932096f4bd3c6518f6d0c508f"}
Nov 01 20:45:53.685368 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.685312    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podStartSLOduration=2.68529836 podStartE2EDuration="2.68529836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:53.682785637 +0000 UTC m=+910.731895455" watchObservedRunningTime="2026-11-01 20:45:53.68529836 +0000 UTC m=+910.734408241"
Nov 01 20:45:53.741676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.741399    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"]
Nov 01 20:45:53.750220 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:45:53.750191    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbec3a4_9f96_41b4_94b5_2fdc0f8e803d.slice/crio-d32bae0c1240c7fdc50cc1f60e130f0245e704037feb24e10d55d1dc8ad408ea WatchSource:0}: Error finding container d32bae0c1240c7fdc50cc1f60e130f0245e704037feb24e10d55d1dc8ad408ea: Status 404 returned error can't find the container with id d32bae0c1240c7fdc50cc1f60e130f0245e704037feb24e10d55d1dc8ad408ea
Nov 01 20:45:53.752579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.752544    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:53.752716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.752629    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:53.752780 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.752735    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:45:53.993250 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.993215    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:45:53.993408 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:53.993271    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:45:54.325688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.324725    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:54.333687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.326020    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:45:54.333687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.326075    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:45:54.581331 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.581235    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" event={"ID":"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d","Type":"ContainerStarted","Data":"d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5"}
Nov 01 20:45:54.581331 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.581298    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" event={"ID":"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d","Type":"ContainerStarted","Data":"d32bae0c1240c7fdc50cc1f60e130f0245e704037feb24e10d55d1dc8ad408ea"}
Nov 01 20:45:54.729680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.728755    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podStartSLOduration=2.7287365230000002 podStartE2EDuration="2.728736523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:54.72720477 +0000 UTC m=+911.776314622" watchObservedRunningTime="2026-11-01 20:45:54.728736523 +0000 UTC m=+911.777846343"
Nov 01 20:45:54.992854 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.992825    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:45:54.993223 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:54.992878    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:45:55.279679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.279237    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:45:55.281800 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.281763    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:45:55.281957 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.281823    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:45:55.320145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.320100    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:45:55.320447 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.320417    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:45:55.991831 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.991797    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:45:55.991993 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:55.991850    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:45:56.275740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:56.275071    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:45:56.275740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:56.275127    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:45:56.322200 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:56.322166    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:45:56.322508 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:56.322463    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:45:56.992438 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:56.992396    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:45:56.992637 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:56.992462    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:45:57.274732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:57.273978    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:45:57.274732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:57.274038    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:45:57.319110 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:57.319071    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:45:57.319723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:57.319687    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:45:57.992101 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:57.992062    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:45:57.992314 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:57.992294    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:45:58.282092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:58.282001    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:45:58.285126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:58.285091    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:45:58.321677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:58.321045    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:45:58.321677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:58.321108    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:45:59.175050 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.175018    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:59.175970 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.175946    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:59.176789 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.176770    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 20:45:59.274425 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.274381    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:45:59.274621 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.274442    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:45:59.529741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.529061    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:59.531204 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.531042    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:45:59.532643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:45:59.532597    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 20:46:00.273851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:00.273787    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:46:00.274329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:00.274212    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:46:01.277293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:01.277265    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:46:01.277689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:01.277643    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:46:01.278390 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:01.278370    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 20:46:23.366172 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:46:23.366129    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:46:52.196525 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.196494    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"]
Nov 01 20:46:52.200616 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.200599    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.268716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.268683    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.268716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.268721    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.268967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.268839    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.268967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.268934    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.268967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.268963    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgww\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.269086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.269036    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.269086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.269078    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.269168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.269102    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.269168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.269127    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.298551 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.298519    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"]
Nov 01 20:46:52.370139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370106    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370139 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370141    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370181    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370217    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370232    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgww\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370282    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370304    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370322    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370353    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370781 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370683    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370837 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370789    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.370902 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.370884    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.371067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.371049    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.371375 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.371358    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.373027 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.373005    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.373210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.373190    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.398137 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.398107    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"]
Nov 01 20:46:52.402261 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.402240    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.431330 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.431293    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.431611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.431588    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgww\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.462870 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.456482    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"distinct-w0-208f2b-kserve-self-signed-certs\""
Nov 01 20:46:52.462870 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.460261    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"]
Nov 01 20:46:52.471829 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.471789    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.471829 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.471828    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.472059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.471998    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.472059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.472043    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.472208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.472183    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4mxz\" (UniqueName: \"kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.472262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.472249    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.515389 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.515353    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:52.573932 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.573879    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.573941    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.574072    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.574103    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.574146    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4mxz\" (UniqueName: \"kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.574200    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574628 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.574378    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.574838 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.574746    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.575226 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.575202    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.579233 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.579208    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.580850 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.580351    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.644424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.644360    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4mxz\" (UniqueName: \"kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz\") pod \"distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.719507 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.719433    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:52.958074 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.958046    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"]
Nov 01 20:46:52.959439 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:46:52.959407    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b71b26_13c2_4d9a_885d_a7c3be6a9a3e.slice/crio-eda350f5e4ede2f7ff5c437cb8bc6edfef12c87679ff4f5c8a89408c92789227 WatchSource:0}: Error finding container eda350f5e4ede2f7ff5c437cb8bc6edfef12c87679ff4f5c8a89408c92789227: Status 404 returned error can't find the container with id eda350f5e4ede2f7ff5c437cb8bc6edfef12c87679ff4f5c8a89408c92789227
Nov 01 20:46:52.962350 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.962316    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:46:52.962455 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.962397    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:46:52.962455 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.962444    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:46:52.998815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:52.998784    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" event={"ID":"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e","Type":"ContainerStarted","Data":"eda350f5e4ede2f7ff5c437cb8bc6edfef12c87679ff4f5c8a89408c92789227"}
Nov 01 20:46:53.094907 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.094869    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"]
Nov 01 20:46:53.098707 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.098690    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.123479 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.123449    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"]
Nov 01 20:46:53.123880 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:46:53.123854    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9583caee_9c57_4cd6_9e29_2765df13ec42.slice/crio-6aec59e7673116741d3bc7a2348f0a33754aa68f1d311d901afe59ad179ea7c3 WatchSource:0}: Error finding container 6aec59e7673116741d3bc7a2348f0a33754aa68f1d311d901afe59ad179ea7c3: Status 404 returned error can't find the container with id 6aec59e7673116741d3bc7a2348f0a33754aa68f1d311d901afe59ad179ea7c3
Nov 01 20:46:53.160828 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.160267    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"]
Nov 01 20:46:53.182224 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.182188    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.182808 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.182779    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183038    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183152 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183068    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183152 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183101    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183152 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183146    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183312 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183172    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183312 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183267    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.183412 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.183336    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvzh\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284322 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284246    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284322 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284291    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284377    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284427    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvzh\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284481    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284542    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284644    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284647    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284697    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.284884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284723    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.285311 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.284977    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.285311 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.285005    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.285311 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.285007    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.285615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.285594    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.287242 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.287221    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.287758 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.287743    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.343942 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.343915    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.344475 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.344448    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvzh\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.393214 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:46:53.387442    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:46:53.413147 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.413108    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:53.855183 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.855151    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"]
Nov 01 20:46:53.856706 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:46:53.856679    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2174bfb3_d995_43a9_9616_495310f38b10.slice/crio-69e4893d1aa60d68d7259b8decc034e5e86ee0e5d6017d30575ea563785647bf WatchSource:0}: Error finding container 69e4893d1aa60d68d7259b8decc034e5e86ee0e5d6017d30575ea563785647bf: Status 404 returned error can't find the container with id 69e4893d1aa60d68d7259b8decc034e5e86ee0e5d6017d30575ea563785647bf
Nov 01 20:46:53.859572 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.859543    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:46:53.859643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.859609    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:46:53.859643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:53.859638    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:46:54.029108 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.029072    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" event={"ID":"9583caee-9c57-4cd6-9e29-2765df13ec42","Type":"ContainerStarted","Data":"80f7526d9c96fbc3e2623a532e163de9d03949b43fb5eb8b3d6df57c0775fa70"}
Nov 01 20:46:54.029293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.029117    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" event={"ID":"9583caee-9c57-4cd6-9e29-2765df13ec42","Type":"ContainerStarted","Data":"6aec59e7673116741d3bc7a2348f0a33754aa68f1d311d901afe59ad179ea7c3"}
Nov 01 20:46:54.031244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.031214    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" event={"ID":"2174bfb3-d995-43a9-9616-495310f38b10","Type":"ContainerStarted","Data":"7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f"}
Nov 01 20:46:54.031408 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.031271    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" event={"ID":"2174bfb3-d995-43a9-9616-495310f38b10","Type":"ContainerStarted","Data":"69e4893d1aa60d68d7259b8decc034e5e86ee0e5d6017d30575ea563785647bf"}
Nov 01 20:46:54.032943 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.032919    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" event={"ID":"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e","Type":"ContainerStarted","Data":"7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb"}
Nov 01 20:46:54.148717 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.148641    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podStartSLOduration=2.148627899 podStartE2EDuration="2.148627899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:54.145939865 +0000 UTC m=+971.195049683" watchObservedRunningTime="2026-11-01 20:46:54.148627899 +0000 UTC m=+971.197737717"
Nov 01 20:46:54.269012 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.268941    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podStartSLOduration=2.268926527 podStartE2EDuration="2.268926527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:54.266897714 +0000 UTC m=+971.316007537" watchObservedRunningTime="2026-11-01 20:46:54.268926527 +0000 UTC m=+971.318036346"
Nov 01 20:46:54.393166 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.393097    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podStartSLOduration=1.39307791 podStartE2EDuration="1.39307791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:54.391871282 +0000 UTC m=+971.440981105" watchObservedRunningTime="2026-11-01 20:46:54.39307791 +0000 UTC m=+971.442187731"
Nov 01 20:46:54.417736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.417356    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:46:54.417736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.417566    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:46:54.417736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.417612    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:46:54.528681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.527943    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:46:54.536682 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.533936    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 20:46:54.536682 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.533997    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 20:46:54.730743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:54.728768    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:46:55.414679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:55.414103    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:46:55.414679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:55.414173    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:46:55.522681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:55.518765    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 20:46:55.522681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:55.518828    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 20:46:56.050761 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.049217    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"]
Nov 01 20:46:56.054461 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.054438    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.149838 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.149679    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"]
Nov 01 20:46:56.153839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.153810    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"distinct-w6-dd038b-kserve-self-signed-certs\""
Nov 01 20:46:56.162239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.162212    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.162452 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.162436    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.162662 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.162622    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxvq\" (UniqueName: \"kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.163487 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.163449    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.163569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.163499    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.163634 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.163606    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265250 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265210    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265490 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265461    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265633 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265610    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265733    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265854    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265967    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.265992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.265979    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.266112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.266100    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxvq\" (UniqueName: \"kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.266256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.266235    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.268220 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.268172    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.269193 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.269087    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.346440 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.345677    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxvq\" (UniqueName: \"kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq\") pod \"distinct-w6-dd038b-kserve-75b476dcb8-pn92g\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.367326 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.367294    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:56.416681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.415077    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:46:56.416681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.415137    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:46:56.565706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.565042    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 20:46:56.565706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.565087    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 20:46:56.845957 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:56.845771    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"]
Nov 01 20:46:56.846955 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:46:56.846922    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode76854fd_0531_48e5_9f07_5592acd1d893.slice/crio-5c53ccf68d5dd251b175459cdbd779531884908bc1b02c4374e64f86aef2c680 WatchSource:0}: Error finding container 5c53ccf68d5dd251b175459cdbd779531884908bc1b02c4374e64f86aef2c680: Status 404 returned error can't find the container with id 5c53ccf68d5dd251b175459cdbd779531884908bc1b02c4374e64f86aef2c680
Nov 01 20:46:57.064328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.064290    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" event={"ID":"e76854fd-0531-48e5-9f07-5592acd1d893","Type":"ContainerStarted","Data":"11b22e144b66fdb31094e88e6e6b417ee860f4063d2d490220dd07fb259b5f68"}
Nov 01 20:46:57.064328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.064336    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" event={"ID":"e76854fd-0531-48e5-9f07-5592acd1d893","Type":"ContainerStarted","Data":"5c53ccf68d5dd251b175459cdbd779531884908bc1b02c4374e64f86aef2c680"}
Nov 01 20:46:57.181391 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.181336    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podStartSLOduration=1.181322294 podStartE2EDuration="1.181322294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:57.180048736 +0000 UTC m=+974.229158556" watchObservedRunningTime="2026-11-01 20:46:57.181322294 +0000 UTC m=+974.230432112"
Nov 01 20:46:57.367537 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.367497    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:46:57.421678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.418115    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:46:57.421678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.418184    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:46:57.520749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.519090    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 20:46:57.520749 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:57.519156    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 20:46:58.437614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:58.421693    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:46:58.437614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:58.421756    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:46:58.518117 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:58.518080    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 20:46:58.518391 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:58.518363    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 20:46:59.425731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:59.425153    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:46:59.425731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:59.425216    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:46:59.519280 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:59.519224    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 20:46:59.519945 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:46:59.519890    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 20:47:00.413944 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:00.413912    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:47:00.414127 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:00.413963    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:47:00.521695 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:00.521639    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:47:00.522213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:00.522195    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:47:00.522941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:00.522916    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 20:47:01.418718 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:01.418687    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:47:01.418979 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:01.418957    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:47:01.419723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:01.419708    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 20:47:02.720084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:02.720048    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:47:02.732265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:02.732231    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:47:03.116454 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:03.116368    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:47:06.367430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:06.367386    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:47:06.380587 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:06.380559    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:47:07.145399 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.145366    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:47:07.565606 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.565519    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"]
Nov 01 20:47:07.569612 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.569591    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.676182 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.676149    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"]
Nov 01 20:47:07.756102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756067    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756102    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756363 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756250    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756363 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756296    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwn4\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756435    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756469    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756496    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756636    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.756725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.756688    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.857673 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.857548    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.857673 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.857620    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwn4\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.857914 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.857746    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.857914 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.857791    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.857923    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858005    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858142 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858040    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858142 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858065    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858125    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858174    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858390 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858368    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858442    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858574    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.858733 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.858715    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.860134 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.860115    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.860371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.860355    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.915156 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.915121    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:07.915439 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:07.915412    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwn4\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:08.188234 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:08.188198    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:08.618421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:08.618337    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"]
Nov 01 20:47:08.660212 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:47:08.660178    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bb5619_4bfe_49fb_a935_d867028f46b5.slice/crio-836f917e1fb4fae41260e796ccd104d8b43cb5fed39cc0bcfc1ab5254fee1466 WatchSource:0}: Error finding container 836f917e1fb4fae41260e796ccd104d8b43cb5fed39cc0bcfc1ab5254fee1466: Status 404 returned error can't find the container with id 836f917e1fb4fae41260e796ccd104d8b43cb5fed39cc0bcfc1ab5254fee1466
Nov 01 20:47:08.662730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:08.662695    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:47:08.662825 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:08.662770    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:47:08.662825 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:08.662802    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:47:09.148035 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:09.147998    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" event={"ID":"f9bb5619-4bfe-49fb-a935-d867028f46b5","Type":"ContainerStarted","Data":"db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243"}
Nov 01 20:47:09.148209 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:09.148058    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" event={"ID":"f9bb5619-4bfe-49fb-a935-d867028f46b5","Type":"ContainerStarted","Data":"836f917e1fb4fae41260e796ccd104d8b43cb5fed39cc0bcfc1ab5254fee1466"}
Nov 01 20:47:09.189338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:09.189311    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:09.190465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:09.190439    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:09.190579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:09.190489    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:09.356455 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:09.356400    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podStartSLOduration=2.356389486 podStartE2EDuration="2.356389486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:09.356224709 +0000 UTC m=+986.405334529" watchObservedRunningTime="2026-11-01 20:47:09.356389486 +0000 UTC m=+986.405499305"
Nov 01 20:47:10.188795 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:10.188763    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:10.189272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:10.188816    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:11.189041 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:11.189009    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:11.189546 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:11.189062    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:12.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:12.189426    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:12.190003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:12.189482    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:13.189267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:13.189230    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:13.189475 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:13.189294    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:14.188626 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:14.188593    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:14.189190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:14.188666    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:15.189254 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:15.189220    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 20:47:15.189802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:15.189286    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 20:47:16.194100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:16.194069    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:16.194540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:16.194382    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:16.195358 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:16.195335    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:47:23.366729 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:47:23.366647    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:47:53.365832 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:47:53.365795    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:47:53.564889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.564855    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"]
Nov 01 20:47:53.568943 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.568924    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664243 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664200    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664452 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664430    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664470    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664608 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664593    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664713    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664742    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664881    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664945 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664932    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq92c\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.664982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.664965    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.766689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.766636    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.766877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.766763    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.766877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.766794    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767163 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767139    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767327 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767127    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767327 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767234    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767366    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq92c\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767397    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767259    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767429    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767784 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767614    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767784 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767649    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.767975 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.767953    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.768523 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.768499    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.770192 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.770169    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.770454 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.770435    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"]
Nov 01 20:47:53.770594 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.770581    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.825848 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.825815    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.825992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.825953    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq92c\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:53.882451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:53.882416    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:54.315011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.314984    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"]
Nov 01 20:47:54.316800 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:47:54.316776    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968359e6_daa8_4326_958f_9aebb9837f10.slice/crio-8d64319873c87da880d94975efcd2ea4a7fd0d5dd4762df764a3764658922b11 WatchSource:0}: Error finding container 8d64319873c87da880d94975efcd2ea4a7fd0d5dd4762df764a3764658922b11: Status 404 returned error can't find the container with id 8d64319873c87da880d94975efcd2ea4a7fd0d5dd4762df764a3764658922b11
Nov 01 20:47:54.327267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.327230    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:47:54.327359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.327302    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:47:54.327359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.327355    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:47:54.446012 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.445980    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" event={"ID":"968359e6-daa8-4326-958f-9aebb9837f10","Type":"ContainerStarted","Data":"3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7"}
Nov 01 20:47:54.446447 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.446060    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" event={"ID":"968359e6-daa8-4326-958f-9aebb9837f10","Type":"ContainerStarted","Data":"8d64319873c87da880d94975efcd2ea4a7fd0d5dd4762df764a3764658922b11"}
Nov 01 20:47:54.566616 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.566514    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podStartSLOduration=1.566502032 podStartE2EDuration="1.566502032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:54.563098393 +0000 UTC m=+1031.612208213" watchObservedRunningTime="2026-11-01 20:47:54.566502032 +0000 UTC m=+1031.615611849"
Nov 01 20:47:54.883317 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:54.883233    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:47:55.883969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:55.883921    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:47:55.884367 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:55.883987    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:47:56.883056 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:56.883017    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:47:56.883325 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:56.883075    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:47:57.883029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:57.882994    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:47:57.883432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:57.883050    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:47:58.882986 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:58.882946    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:47:58.883170 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:58.883001    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:47:59.882946 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:59.882911    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:47:59.883120 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:47:59.882967    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:48:01.117944 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:01.117906    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:48:01.118441 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:01.118256    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:48:01.119361 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:01.119339    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 20:48:23.366686 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:48:23.366618    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:48:44.342895 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.342860    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"]
Nov 01 20:48:44.346802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.346786    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.434982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.434951    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"]
Nov 01 20:48:44.442743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.442715    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.442870 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.442778    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.442956 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.442933    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.443011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.442971    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.443092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.443078    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9gjt\" (UniqueName: \"kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.443222 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.443207    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.451202 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.451177    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-208f2b\"/\"distinct-2-w0-208f2b-kserve-self-signed-certs\""
Nov 01 20:48:44.543953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.543916    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.543990    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544206 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544159    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544199    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544304 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544251    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9gjt\" (UniqueName: \"kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544429 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544412    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544522 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544509    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544570    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.544773 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.544753    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.546412 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.546373    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.546492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.546455    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.602388 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.602321    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9gjt\" (UniqueName: \"kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt\") pod \"distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") " pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:44.659548 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:44.659513    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:45.041067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:45.041033    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"]
Nov 01 20:48:45.042481 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:48:45.042447    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4933e452_e3a8_4ba6_9bfa_b4d07c2acba5.slice/crio-ee7f65eb5dda4a7f95257b930d62d71ced4e7f52f567e6fb184ca8aaf37cb059 WatchSource:0}: Error finding container ee7f65eb5dda4a7f95257b930d62d71ced4e7f52f567e6fb184ca8aaf37cb059: Status 404 returned error can't find the container with id ee7f65eb5dda4a7f95257b930d62d71ced4e7f52f567e6fb184ca8aaf37cb059
Nov 01 20:48:45.790378 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:45.790339    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" event={"ID":"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5","Type":"ContainerStarted","Data":"08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803"}
Nov 01 20:48:45.790856 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:45.790389    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" event={"ID":"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5","Type":"ContainerStarted","Data":"ee7f65eb5dda4a7f95257b930d62d71ced4e7f52f567e6fb184ca8aaf37cb059"}
Nov 01 20:48:45.908244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:45.908192    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podStartSLOduration=1.908178028 podStartE2EDuration="1.908178028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:48:45.906219342 +0000 UTC m=+1082.955329166" watchObservedRunningTime="2026-11-01 20:48:45.908178028 +0000 UTC m=+1082.957287845"
Nov 01 20:48:46.659602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:46.659571    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:53.368817 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:48:53.368761    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:48:54.659719 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:54.659684    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:54.672435 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:54.672410    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:48:54.853092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:48:54.853066    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:49:23.365881 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:49:23.365840    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:49:23.388764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.388728    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"]
Nov 01 20:49:23.397631 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.397603    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.491719 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.491680    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.491921 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.491811    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.491921 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.491835    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcf82\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.492092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.491974    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.492092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.492010    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.492092 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.492047    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.492239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.492120    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.492239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.492184    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.492320 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.492272    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.593734 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.593696    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.593734 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.593736    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.593734 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.593751    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.593814    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.593853    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.593907    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594038    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594289 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594122    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594289 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594153    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcf82\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594467 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594426    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594728 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594459    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"]
Nov 01 20:49:23.594883 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594841    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.594997 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.594974    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.595215 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.595197    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.595272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.595203    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.597716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.597685    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.598184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.598164    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.651646 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.651615    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.651858 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.651750    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcf82\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:23.711891 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:23.711855    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:24.145903 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.145875    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"]
Nov 01 20:49:24.147744 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:49:24.147717    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3064070_bfb9_4afe_b20c_20127131740a.slice/crio-a89732bb84b28423e4bd338f1af45ab8d34c18b441a974663b1de3f46a768070 WatchSource:0}: Error finding container a89732bb84b28423e4bd338f1af45ab8d34c18b441a974663b1de3f46a768070: Status 404 returned error can't find the container with id a89732bb84b28423e4bd338f1af45ab8d34c18b441a974663b1de3f46a768070
Nov 01 20:49:24.150428 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.150383    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:24.150500 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.150473    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:24.150536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.150502    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:24.552483 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.552455    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"]
Nov 01 20:49:24.556582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.556550    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607194    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lht92\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607256    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607293    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607321    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607346    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607450    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607495    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607519    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.607649 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.607639    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.639681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.623787    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"]
Nov 01 20:49:24.709572 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709525    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709595    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709629    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709675    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709793    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709849    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.709876    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.710015    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710474 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.710249    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710524 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.710486    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.710629 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.710612    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lht92\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.711310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.711282    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.711802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.711780    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.711896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.711871    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.713762 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.713740    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.714084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.714066    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.768209 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.768179    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.768359 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.768248    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lht92\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:24.871979 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:24.871890    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:25.036713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.036677    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" event={"ID":"f3064070-bfb9-4afe-b20c-20127131740a","Type":"ContainerStarted","Data":"9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d"}
Nov 01 20:49:25.036912 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.036732    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" event={"ID":"f3064070-bfb9-4afe-b20c-20127131740a","Type":"ContainerStarted","Data":"a89732bb84b28423e4bd338f1af45ab8d34c18b441a974663b1de3f46a768070"}
Nov 01 20:49:25.165043 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.164983    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podStartSLOduration=2.16496418 podStartE2EDuration="2.16496418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:25.160387256 +0000 UTC m=+1122.209497074" watchObservedRunningTime="2026-11-01 20:49:25.16496418 +0000 UTC m=+1122.214073999"
Nov 01 20:49:25.329081 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.329052    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"]
Nov 01 20:49:25.330619 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:49:25.330589    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cbb5f2_8cbf_44e2_b5c0_8f54cb6bb8fa.slice/crio-0e88c324d864bec7996739d7fa8ae8710bcca17acf94013eb220e619e6f56792 WatchSource:0}: Error finding container 0e88c324d864bec7996739d7fa8ae8710bcca17acf94013eb220e619e6f56792: Status 404 returned error can't find the container with id 0e88c324d864bec7996739d7fa8ae8710bcca17acf94013eb220e619e6f56792
Nov 01 20:49:25.333320 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.333282    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:25.333423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.333361    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:25.333423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.333393    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:25.719164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.716828    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:25.720356 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.720326    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:25.720582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:25.720547    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:26.050735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.047746    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" event={"ID":"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa","Type":"ContainerStarted","Data":"629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12"}
Nov 01 20:49:26.050735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.047847    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" event={"ID":"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa","Type":"ContainerStarted","Data":"0e88c324d864bec7996739d7fa8ae8710bcca17acf94013eb220e619e6f56792"}
Nov 01 20:49:26.180842 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.180792    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podStartSLOduration=2.180777897 podStartE2EDuration="2.180777897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:26.178611087 +0000 UTC m=+1123.227720905" watchObservedRunningTime="2026-11-01 20:49:26.180777897 +0000 UTC m=+1123.229887715"
Nov 01 20:49:26.725677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.725118    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:26.725677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.725212    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:26.872945 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.872873    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:26.874171 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.874150    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:49:26.874239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:26.874198    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:49:27.713680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:27.713102    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:27.713680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:27.713177    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:27.873179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:27.873137    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:49:27.873691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:27.873205    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:49:28.714120 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:28.714080    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:28.714385 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:28.714358    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:28.873118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:28.873081    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:49:28.873387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:28.873363    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:49:29.715155 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:29.715111    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:29.715466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:29.715434    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:29.872394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:29.872355    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:49:29.872576 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:29.872422    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:49:30.367829 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.367796    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"]
Nov 01 20:49:30.372000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.371980    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.415332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.414008    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.415332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.414081    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.415332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.414221    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.415332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.414258    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.415332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.414283    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2g7\" (UniqueName: \"kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.415332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.414354    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.465229 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.459979    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"]
Nov 01 20:49:30.564691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.508105    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"distinct-2-w4-1a1e9e-kserve-self-signed-certs\""
Nov 01 20:49:30.564691 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.522522    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.565058 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.565029    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.572014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.571972    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.582805 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.582764    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.588790 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.588749    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2g7\" (UniqueName: \"kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.670709 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.522975    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.670709 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.663702    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.670709 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.664164    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.670709 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.664927    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.670709 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.667901    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.680183 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.680156    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.740683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.716130    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:30.740683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.716190    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:30.747141 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.747041    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2g7\" (UniqueName: \"kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7\") pod \"distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:30.873222 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.873185    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:49:30.873404 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.873241    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:49:30.984961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:30.984872    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:31.475889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:31.475120    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"]
Nov 01 20:49:31.477033 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:49:31.477003    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97907b4a_9475_4f3f_a0ad_cf6501ee24e0.slice/crio-0ca235d71d337d9a169ba4aff2b75c0b12e6511d79825a4c2d9cdbac945a6d66 WatchSource:0}: Error finding container 0ca235d71d337d9a169ba4aff2b75c0b12e6511d79825a4c2d9cdbac945a6d66: Status 404 returned error can't find the container with id 0ca235d71d337d9a169ba4aff2b75c0b12e6511d79825a4c2d9cdbac945a6d66
Nov 01 20:49:31.712946 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:31.712904    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:49:31.713125 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:31.712975    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:49:31.873184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:31.873085    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:49:31.873451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:31.873418    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:49:32.103575 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.103529    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" event={"ID":"97907b4a-9475-4f3f-a0ad-cf6501ee24e0","Type":"ContainerStarted","Data":"56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab"}
Nov 01 20:49:32.103575 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.103576    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" event={"ID":"97907b4a-9475-4f3f-a0ad-cf6501ee24e0","Type":"ContainerStarted","Data":"0ca235d71d337d9a169ba4aff2b75c0b12e6511d79825a4c2d9cdbac945a6d66"}
Nov 01 20:49:32.219308 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.219249    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podStartSLOduration=2.219235306 podStartE2EDuration="2.219235306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:32.218796769 +0000 UTC m=+1129.267906589" watchObservedRunningTime="2026-11-01 20:49:32.219235306 +0000 UTC m=+1129.268345124"
Nov 01 20:49:32.717132 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.717100    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:32.717526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.717500    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:32.718331 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.718310    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 20:49:32.890240 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.890191    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:32.890742 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.890718    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:32.891531 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.891515    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 20:49:32.986568 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:32.986488    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:40.986180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:40.986142    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:40.998260 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:40.998230    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:41.176231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:41.176199    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:49:48.952821 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:48.952774    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:49:49.553845 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:49.553781    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:49:53.367112 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:49:53.367077    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:49:54.118977 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.118948    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"]
Nov 01 20:49:54.123770 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.123753    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.156206 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.156180    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcf6\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.156484 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.156452    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.156744 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.156702    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.156846 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.156761    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.156913 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.156882    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.157090 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.157072    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.157314 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.157291    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.157410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.157325    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.157410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.157395    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258233    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258281    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcf6\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258389    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258418    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258435    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258632 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258495    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258708 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258639    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258740    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258794 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258765    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.258869 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.258851    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.259059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.259027    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.259059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.259046    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.259212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.259070    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.259597 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.259571    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.260972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.260950    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.261271 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.261250    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.318713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.318438    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.318713 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.318447    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcf6\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.324343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.324311    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"]
Nov 01 20:49:54.437925 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.437899    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:54.866937 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.866910    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"]
Nov 01 20:49:54.867612 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:49:54.867585    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c247f1_be8b_42e2_8db0_2a0e3ba95df8.slice/crio-3a067d293370de545b91404f0f6041a8da68d8f0af312272b0ccee83ab3f5ee0 WatchSource:0}: Error finding container 3a067d293370de545b91404f0f6041a8da68d8f0af312272b0ccee83ab3f5ee0: Status 404 returned error can't find the container with id 3a067d293370de545b91404f0f6041a8da68d8f0af312272b0ccee83ab3f5ee0
Nov 01 20:49:54.869877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.869849    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:54.869942 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.869913    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:54.869977 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:54.869941    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:49:55.304281 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:55.304240    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" event={"ID":"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8","Type":"ContainerStarted","Data":"aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37"}
Nov 01 20:49:55.304435 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:55.304305    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" event={"ID":"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8","Type":"ContainerStarted","Data":"3a067d293370de545b91404f0f6041a8da68d8f0af312272b0ccee83ab3f5ee0"}
Nov 01 20:49:55.438791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:55.438760    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:49:55.439912 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:55.439888    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:49:55.439969 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:55.439938    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:49:55.485314 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:55.485256    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podStartSLOduration=1.485236003 podStartE2EDuration="1.485236003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:55.478589705 +0000 UTC m=+1152.527699546" watchObservedRunningTime="2026-11-01 20:49:55.485236003 +0000 UTC m=+1152.534345823"
Nov 01 20:49:56.438500 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:56.438469    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:49:56.438708 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:56.438518    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:49:57.438466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:57.438429    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:49:57.438918 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:57.438479    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:49:58.439204 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:58.439167    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:49:58.439618 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:58.439223    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:49:59.439157 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:59.439124    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:49:59.439325 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:49:59.439171    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:50:00.438536 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:00.438504    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:50:00.438725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:00.438553    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:50:01.438908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:01.438874    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:50:01.439413 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:01.438932    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:50:02.442995 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:02.442965    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:50:02.443394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:02.443374    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:50:02.444091 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:02.444065    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 20:50:23.365829 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:50:23.365794    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:50:25.692721 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.692691    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"]
Nov 01 20:50:25.697764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.697747    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732033    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732069    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732165    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732240    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732327 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732291    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732360    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmxq\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732412    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732458 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732433    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.732522 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.732497    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833336    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833546 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833409    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833546 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833480    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmxq\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833546 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833538    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833564    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833596    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833852 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833757    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833852 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833794    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.833953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833939    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.834008 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.833985    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.834157 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.834135    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.834238 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.834172    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.834388 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.834368    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.834730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.834711    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.835971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.835946    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.836585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.836561    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.890519 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.890493    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.890634 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.890618    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmxq\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:25.900102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:25.900080    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"]
Nov 01 20:50:26.012764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.012700    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:26.440598 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.440572    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"]
Nov 01 20:50:26.441911 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:50:26.441873    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70c2a48_02a0_47ab_8d9c_52b3d3f590a0.slice/crio-2b2374418083ffc00683f30713475f327d1cc74ed1ad6550501f5aab2d3aba83 WatchSource:0}: Error finding container 2b2374418083ffc00683f30713475f327d1cc74ed1ad6550501f5aab2d3aba83: Status 404 returned error can't find the container with id 2b2374418083ffc00683f30713475f327d1cc74ed1ad6550501f5aab2d3aba83
Nov 01 20:50:26.444105 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.444074    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:50:26.444184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.444135    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:50:26.444184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.444161    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:50:26.533093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.533062    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" event={"ID":"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0","Type":"ContainerStarted","Data":"73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8"}
Nov 01 20:50:26.533215 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:26.533120    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" event={"ID":"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0","Type":"ContainerStarted","Data":"2b2374418083ffc00683f30713475f327d1cc74ed1ad6550501f5aab2d3aba83"}
Nov 01 20:50:27.012896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:27.012866    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:27.014158 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:27.014134    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:27.014273 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:27.014182    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:28.014679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:28.013162    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:28.014679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:28.013240    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:29.018714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:29.013172    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:29.018714 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:29.013248    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:30.016688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:30.015996    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:30.016688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:30.016053    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:31.017048 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:31.017011    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:31.017610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:31.017576    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:32.019101 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:32.019061    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:32.019562 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:32.019125    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:33.013558 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:33.013520    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 20:50:33.013735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:33.013587    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 20:50:34.017353 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:34.017325    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:34.017791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:34.017697    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:34.018347 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:34.018326    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:50:34.185754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:34.185711    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podStartSLOduration=9.18569651 podStartE2EDuration="9.18569651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:26.662017623 +0000 UTC m=+1183.711127447" watchObservedRunningTime="2026-11-01 20:50:34.18569651 +0000 UTC m=+1191.234806328"
Nov 01 20:50:36.549678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:36.548756    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 01 20:50:36.612883 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:36.611885    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:36.747796 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:36.746753    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:36.846862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:36.846802    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:36.896705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:36.895845    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.073625 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.073238    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 01 20:50:37.139789 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.139744    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.154860 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.154822    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.156016 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.155977    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:50:37.168973 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.168934    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.169412 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.169381    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.279679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.278639    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.356008 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.355906    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 20:50:37.378846 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.378739    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.405586 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.404848    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.550727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.549738    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.550727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.550147    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:50:37.616272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.613734    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.616272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.613887    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:50:37.790679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.747089    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.790679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.747480    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:50:37.848695 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.848275    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:37.848695 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:37.848689    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:50:38.020168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.002994    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:38.020168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.003351    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:50:38.193801 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.193760    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.199919    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.200204    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.200395    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.200705    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.200765    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.200803    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.200838    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:50:38.241593 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.239556    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:50:38.294677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.281924    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:50:38.294677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.282154    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:50:38.466697 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.466626    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 20:50:38.558729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.474253    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:50:38.558729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.546078    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:50:38.558729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.546121    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:50:38.558729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.546151    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:50:38.558729 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.546191    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:50:38.590862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.589228    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:50:38.611734 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.611565    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:50:38.611922 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.611837    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:50:38.629170 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.629129    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:50:38.629393 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.629382    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:50:38.634373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:38.634353    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:50:43.234634 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:43.234603    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:50:43.235968 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:43.235944    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:50:43.246910 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:43.246887    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:50:43.248614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:50:43.247998    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:50:53.366724 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:50:53.366687    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:51:09.427349 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.427269    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"]
Nov 01 20:51:09.431399 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.431381    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.515647 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.515620    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.515810 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.515705    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.515899 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.515875    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.515960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.515915    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkxf\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.516026 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.515973    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.516158 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.516134    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.516207 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.516180    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.516244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.516219    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.516295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.516256    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618100 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618072    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618102    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkxf\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618127    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618306    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618353    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618411    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618462    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618479    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618631 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618618    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618677    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618680    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618842    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.618953 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.618908    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.619121 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.619008    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.621252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.621226    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.623060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.623036    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.629515 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.629408    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"]
Nov 01 20:51:09.748730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.747253    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"]
Nov 01 20:51:09.753342 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.753114    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkxf\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.753342 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.753307    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.754763 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.753706    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:09.785313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.783967    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"]
Nov 01 20:51:09.852637 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.852598    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.852843 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.852768    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.852843 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.852820    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.852961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.852854    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbhq\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.852961 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.852948    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.853069 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.853051    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.853122 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.853106    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.853180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.853137    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.853180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.853172    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954629 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954596    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954643    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954709    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbhq\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954786    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954878    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954931    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.954982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.954959    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955117    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955124    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955165    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955203    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955415    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955764 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955745    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.955950 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.955935    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.960687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.957862    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:09.960687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:09.958116    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:10.012083 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.012024    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:10.012409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.012392    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbhq\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:10.067685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.067210    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:10.133679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.131783    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:10.372269 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.370345    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"]
Nov 01 20:51:10.421169 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.417059    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.516246 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.514538    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"]
Nov 01 20:51:10.533357 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533263    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.533357 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533321    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.533745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533414    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.533745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533467    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.533745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533588    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6b7\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.533745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533673    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.533745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533723    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.534030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533809    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.534030 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.533872    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.637560    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.637605    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.637716    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.637763    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.637885    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6b7\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.637960    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.638010    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.638052    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.644168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.638108    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.664369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.662492    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.675544 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.672960    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.675544 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.673238    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.691812 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.689834    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:10.951676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:10.937073    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:11.073679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.070798    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"]
Nov 01 20:51:11.192453 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.179920    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:11.192453 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.180003    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:11.192453 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.180049    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:11.202779 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.202685    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:11.317198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.317118    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:11.321686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.320264    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:11.434699 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:51:11.412817    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb7e387_fb56_4ea9_b83f_a00bdeaa3c03.slice/crio-45934766e56674213430ef8ae4ee1eca681de360b54d97a4c5856edb7e3fd7ef WatchSource:0}: Error finding container 45934766e56674213430ef8ae4ee1eca681de360b54d97a4c5856edb7e3fd7ef: Status 404 returned error can't find the container with id 45934766e56674213430ef8ae4ee1eca681de360b54d97a4c5856edb7e3fd7ef
Nov 01 20:51:11.434699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.418162    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6b7\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:11.434699 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.424270    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:51:11.464741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.445503    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:11.464741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.445603    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:11.464741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.445648    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:11.464741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.448673    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"]
Nov 01 20:51:11.525877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.503732    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 01 20:51:11.543019 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.542980    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:11.595789 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.595745    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:11.656033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.642896    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:11.656033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.643252    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 20:51:11.656033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.643542    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 01 20:51:11.685831 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.673734    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 20:51:11.699697 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.695103    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:11.991689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:11.988385    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.150740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.140382    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.165889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.165780    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" event={"ID":"f2c28b06-dce7-47b3-8b9b-2501d63f7463","Type":"ContainerStarted","Data":"a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326"}
Nov 01 20:51:12.165889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.165840    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" event={"ID":"f2c28b06-dce7-47b3-8b9b-2501d63f7463","Type":"ContainerStarted","Data":"bcc90010e0656ca879622fe4663f9fff7f07cb1eea1fcee988893a920bc8e0b4"}
Nov 01 20:51:12.179679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.179543    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" event={"ID":"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03","Type":"ContainerStarted","Data":"45934766e56674213430ef8ae4ee1eca681de360b54d97a4c5856edb7e3fd7ef"}
Nov 01 20:51:12.244455 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.243723    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.463678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.454873    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.463678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.456171    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:51:12.490553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.487084    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.559679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.517706    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.695288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.569117    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.695288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.587888    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:51:12.695288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.592093    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podStartSLOduration=3.592076213 podStartE2EDuration="3.592076213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:12.538358175 +0000 UTC m=+1229.587467996" watchObservedRunningTime="2026-11-01 20:51:12.592076213 +0000 UTC m=+1229.641186031"
Nov 01 20:51:12.695288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.617212    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.712366 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.712339    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:51:12.715003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.712739    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.718397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.718367    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.752685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.713713    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:12.752685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.752357    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:51:12.755382 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.753546    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:51:12.755382 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.753600    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:51:12.755382 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.753633    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:51:12.762745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.757704    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:51:12.776460 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.768960    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:51:12.786948 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.786924    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:51:12.801678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.795210    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:51:12.808937 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:12.808913    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:51:13.059933 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.059905    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"]
Nov 01 20:51:13.063862 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:51:13.063827    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61510c86_9623_4b53_8682_da5f35e720a2.slice/crio-026d9eeb79bc172616b52a5805b0750a80333d0a03324604926e33622d475cf8 WatchSource:0}: Error finding container 026d9eeb79bc172616b52a5805b0750a80333d0a03324604926e33622d475cf8: Status 404 returned error can't find the container with id 026d9eeb79bc172616b52a5805b0750a80333d0a03324604926e33622d475cf8
Nov 01 20:51:13.066131 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.066097    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:13.066244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.066172    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:13.066244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.066213    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:13.140240 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.140208    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:13.140436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.140337    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:13.140436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.140380    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:13.276045 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.276013    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" event={"ID":"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03","Type":"ContainerStarted","Data":"f64ce4e70706638491d92cf3ab7f1e9e51128bbac27040423a873c4f42362cb1"}
Nov 01 20:51:13.277930 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.277908    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" event={"ID":"61510c86-9623-4b53-8682-da5f35e720a2","Type":"ContainerStarted","Data":"026d9eeb79bc172616b52a5805b0750a80333d0a03324604926e33622d475cf8"}
Nov 01 20:51:13.986676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:13.983341    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podStartSLOduration=4.983322704 podStartE2EDuration="4.983322704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:13.980591375 +0000 UTC m=+1231.029701192" watchObservedRunningTime="2026-11-01 20:51:13.983322704 +0000 UTC m=+1231.032432523"
Nov 01 20:51:14.079678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.072444    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:14.079678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.073987    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:51:14.079678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.074036    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:51:14.160688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.156330    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:14.160688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.156387    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:14.392871 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.392774    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" event={"ID":"61510c86-9623-4b53-8682-da5f35e720a2","Type":"ContainerStarted","Data":"ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934"}
Nov 01 20:51:14.551777 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.551729    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:14.569861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.569815    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:51:14.570166 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.570117    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:51:14.580978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:14.580738    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podStartSLOduration=4.58072467 podStartE2EDuration="4.58072467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:14.579474404 +0000 UTC m=+1231.628584215" watchObservedRunningTime="2026-11-01 20:51:14.58072467 +0000 UTC m=+1231.629834487"
Nov 01 20:51:15.071681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:15.071055    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:51:15.071681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:15.071120    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:51:15.152685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:15.152015    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:15.152685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:15.152075    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:15.550369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:15.550325    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:51:15.550568 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:15.550387    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:51:16.076424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:16.075011    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:51:16.076424 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:16.075074    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:51:16.136678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:16.136562    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:16.136678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:16.136617    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:16.552680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:16.551024    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:51:16.552680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:16.551085    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:51:17.079680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:17.079134    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:51:17.079680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:17.079193    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:51:17.133681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:17.133027    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:17.133681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:17.133089    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:17.555723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:17.552180    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:51:17.555723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:17.552241    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:51:18.069831 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:18.069794    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:51:18.070135 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:18.070104    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:51:18.177689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:18.171488    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:18.177689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:18.171541    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:18.549081 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:18.549046    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:51:18.549387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:18.549352    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:51:19.068463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:19.068419    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:51:19.068623 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:19.068493    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:51:19.136936 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:19.136902    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:51:19.137193 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:19.137164    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:51:19.544381 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:19.544344    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:51:19.544989 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:19.544955    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:51:20.163697 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:20.163471    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:20.164452 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:20.164431    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:20.167582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:20.167559    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 20:51:20.256047 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:20.256014    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:20.257558 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:20.257163    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:20.258895 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:20.258874    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 20:51:21.015535 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:21.015497    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:21.016148 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:21.016126    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:21.017797 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:21.017772    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 20:51:23.366834 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:51:23.366791    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:51:26.158311 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.158279    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"]
Nov 01 20:51:26.163571 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.163542    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.253442 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.253395    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"]
Nov 01 20:51:26.255392 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255363    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255627    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255723    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255745    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255773    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255799    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255835    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbcc\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.255972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.255921    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.256135 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.256033    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357311    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357538 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357402    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357538 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357445    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357538 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357515    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357549    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357581    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357613    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357727 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357650    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbcc\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.357915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.357755    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.358042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.358018    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.358214 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.358058    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.358292 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.358078    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.358292 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.358128    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.358441 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.358406    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.360256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.360230    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.360677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.360637    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.415446 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.415393    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.415736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.415716    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbcc\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.477592 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.477551    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:26.919500 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.919464    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"]
Nov 01 20:51:26.919500 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:51:26.919482    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bfa8d1_cccb_4d3c_b291_bd77f5d86ef6.slice/crio-550e1034563e5ee7dc84675fc44dd30fc7a3b2b81c86393562fc00e980e5a1c8 WatchSource:0}: Error finding container 550e1034563e5ee7dc84675fc44dd30fc7a3b2b81c86393562fc00e980e5a1c8: Status 404 returned error can't find the container with id 550e1034563e5ee7dc84675fc44dd30fc7a3b2b81c86393562fc00e980e5a1c8
Nov 01 20:51:26.922623 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.922594    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:26.922730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.922675    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:26.922730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:26.922705    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:51:27.579564 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:27.579525    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" event={"ID":"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6","Type":"ContainerStarted","Data":"ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26"}
Nov 01 20:51:27.580086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:27.579580    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" event={"ID":"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6","Type":"ContainerStarted","Data":"550e1034563e5ee7dc84675fc44dd30fc7a3b2b81c86393562fc00e980e5a1c8"}
Nov 01 20:51:27.807683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:27.805514    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podStartSLOduration=1.805495603 podStartE2EDuration="1.805495603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:27.804059118 +0000 UTC m=+1244.853168935" watchObservedRunningTime="2026-11-01 20:51:27.805495603 +0000 UTC m=+1244.854605423"
Nov 01 20:51:28.478263 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:28.478232    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:28.479875 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:28.479836    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 20:51:28.480044 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:28.479892    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 20:51:29.478211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:29.478180    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 20:51:29.478574 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:29.478228    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 20:51:30.478686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:30.478637    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 20:51:30.479150 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:30.478712    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 20:51:31.478687 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:31.478639    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 20:51:31.479089 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:31.478708    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 20:51:32.478369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:32.478335    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 20:51:32.478533 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:32.478397    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 20:51:33.478388 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:33.478355    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 20:51:33.478887 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:33.478426    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 20:51:34.482643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:34.482611    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:34.483297 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:34.483273    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:34.484006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:34.483987    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 20:51:49.204067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.190759    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.373706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.364746    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.508867 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.508754    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.521868 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.521818    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.522535 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.522502    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 20:51:49.608999 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.608939    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:51:49.610224 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.610189    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.654806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.654752    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.783739 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.780756    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.813232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.811039    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:51:49.813232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.811439    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 20:51:49.813232 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.811752    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:49.975846 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:49.975799    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.158681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.157828    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 01 20:51:50.194690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.193752    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.194690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.194218    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:51:50.255684 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.254732    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.337693 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.334232    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": context deadline exceeded"
Nov 01 20:51:50.412357 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.366713    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.412357 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.367168    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:51:50.530680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.517745    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.530680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.518532    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:51:50.555680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.546721    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.555680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.547077    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:51:50.555680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.547381    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:51:50.555680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.547437    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:51:50.709731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.708031    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.709731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.708511    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:51:50.709731 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.709475    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 20:51:50.710016 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.709741    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:51:50.745677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.740731    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.745875 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.745786    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:51:50.798678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.798223    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:50.802677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.799272    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:51:50.826680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.814131    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 20:51:50.826680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.814427    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:51:50.826680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.814594    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 20:51:50.826680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.814825    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:51:50.826680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.814880    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:51:50.826680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:50.814915    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:51:51.341577 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.251870    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:51.341577 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.339494    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:51.371018 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.370988    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:51:51.374786 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.374741    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:51.538827 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.538786    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded"
Nov 01 20:51:51.600557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.600456    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 20:51:51.629741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.601806    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:51.731680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.731386    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:51:51.731881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.731731    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 20:51:51.746992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.740451    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 01 20:51:51.752880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.749413    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:51.960732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.813704    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:51.975131 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.839835    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 20:51:51.990680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.839887    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:51:51.990680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:51.840084    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.037680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.017984    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Nov 01 20:51:52.088790 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.088750    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:51:52.089060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.089028    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.283771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.281993    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.484509 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.484469    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 20:51:52.738007 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.712095    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.750611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.712984    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.757123 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.713623    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 20:51:52.757638 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.727332    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.758071 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.738261    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:52.761248 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.738294    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:51:52.761984 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:52.750237    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 20:51:53.003648 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.003545    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 01 20:51:53.016169 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.016127    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 20:51:53.049035 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.043344    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:51:53.049567 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.043380    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 20:51:53.050043 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.043677    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:53.307787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.301198    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:51:53.479679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.459832    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:51:53.479679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.461303    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:51:53.479679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.461381    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:51:53.479679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.461436    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:51:53.576242 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.576155    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:51:53.582722 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.580516    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:51:53.598000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.597970    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:51:53.599026 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.598995    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:51:53.602232 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:51:53.602205    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:51:53.606220 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.606197    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:51:53.619605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.608153    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:51:53.619605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.609127    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:51:53.619605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.613499    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:51:53.619605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:53.619233    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:51:54.876363 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:54.876331    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"]
Nov 01 20:51:54.876833 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:51:54.876519    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy" containerID="cri-o://27f46f3be09257d3fd46767d380297de8d578a20bdceea32f048f7104639c2e4" gracePeriod=30
Nov 01 20:52:00.058913 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.058883    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"]
Nov 01 20:52:00.059637 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.059365    4872 generic.go:356] "Generic (PLEG): container finished" podID="ada7e173-1bca-4229-becb-69e9edc1250f" containerID="27f46f3be09257d3fd46767d380297de8d578a20bdceea32f048f7104639c2e4" exitCode=0
Nov 01 20:52:00.066456 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.065717    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.066866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.065600    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" event={"ID":"ada7e173-1bca-4229-becb-69e9edc1250f","Type":"ContainerDied","Data":"27f46f3be09257d3fd46767d380297de8d578a20bdceea32f048f7104639c2e4"}
Nov 01 20:52:00.122023 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.121989    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"]
Nov 01 20:52:00.158637 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.158610    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.158991 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.158957    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.159129 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.159113    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.159213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.159137    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.159213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.159156    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.159389 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.159369    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.159720 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.159650    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.159939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.159916    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.160061 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.160040    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdf2\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.168410 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.168376    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0ef08b4-openshift-default-dockercfg-wxggn\""
Nov 01 20:52:00.180372 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.180353    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:52:00.261249 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261217    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261287    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261345    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261418    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261464    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261718 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261499    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:52:00.261718 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261616    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261848 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261726    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261848 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261774    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:52:00.261848 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261822    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.261848 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261824    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:52:00.262054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.261883    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo\") pod \"ada7e173-1bca-4229-becb-69e9edc1250f\" (UID: \"ada7e173-1bca-4229-becb-69e9edc1250f\") "
Nov 01 20:52:00.262054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262028    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:52:00.262054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262046    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:52:00.262265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262236    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.262348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262300    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.262465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262373    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.262465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262417    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdf2\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.262581 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262529    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.262641 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262630    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262643    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.262966    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263045    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263051    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263346    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263109    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263404    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263428    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263544    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263563    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263578    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada7e173-1bca-4229-becb-69e9edc1250f-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263594    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.263611    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.264543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.264224    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:00.265483 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.264815    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "kube-api-access-hnkwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:00.265483 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.264845    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:52:00.266288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.266265    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.266582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.266558    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo" pod "ada7e173-1bca-4229-becb-69e9edc1250f" (UID: "ada7e173-1bca-4229-becb-69e9edc1250f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:52:00.267612 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.267374    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.321312 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.321277    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.321506 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.321321    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdf2\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2\") pod \"e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") " pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.366310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.366211    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada7e173-1bca-4229-becb-69e9edc1250f-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.366310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.366252    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkwl\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-kube-api-access-hnkwl\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.366310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.366268    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada7e173-1bca-4229-becb-69e9edc1250f-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.366310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.366281    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada7e173-1bca-4229-becb-69e9edc1250f-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:52:00.384206 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.384170    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:00.849886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.849856    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"]
Nov 01 20:52:00.852564 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:52:00.852533    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ff28c9_a1d1_4ab8_9a2d_56be971bd8d9.slice/crio-e635eda13c748e1126aabe8f196f86f6edd9e7698d8a7a3052e1d15e0dff6943 WatchSource:0}: Error finding container e635eda13c748e1126aabe8f196f86f6edd9e7698d8a7a3052e1d15e0dff6943: Status 404 returned error can't find the container with id e635eda13c748e1126aabe8f196f86f6edd9e7698d8a7a3052e1d15e0dff6943
Nov 01 20:52:00.855562 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.855527    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:00.855685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.855609    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:00.855685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.855672    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:00.898102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.898075    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"]
Nov 01 20:52:00.899036 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.899015    4872 cpu_manager.go:414] "containerMap: removing container" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy"
Nov 01 20:52:00.899138 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.899037    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy"
Nov 01 20:52:00.899252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.899239    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" containerName="istio-proxy"
Nov 01 20:52:00.904507 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.904488    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.977361 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.970018    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"]
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978079    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978129    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978244    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978365    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978494    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978771    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978850    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.978935    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:00.979939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:00.979043    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkc5\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.029546 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.029506    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d0d57533-openshift-default-dockercfg-kcrtj\""
Nov 01 20:52:01.080137 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080098    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080267    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080317    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080363    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080427    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkc5\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080469    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080496    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.080615 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080546    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.081005 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.080621    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.082040 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.082005    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.082877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.082845    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.083083 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.083028    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.083083 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.083029    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.083225 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.083092    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.083306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.083277    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.087987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.085941    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" event={"ID":"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9","Type":"ContainerStarted","Data":"46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"}
Nov 01 20:52:01.087987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.085985    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" event={"ID":"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9","Type":"ContainerStarted","Data":"e635eda13c748e1126aabe8f196f86f6edd9e7698d8a7a3052e1d15e0dff6943"}
Nov 01 20:52:01.087987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.086371    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.087987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.087173    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4" event={"ID":"ada7e173-1bca-4229-becb-69e9edc1250f","Type":"ContainerDied","Data":"e055c52472a4d513fc77fab7ea93b9e5a60d84852eb0b530dcd23bf18517c082"}
Nov 01 20:52:01.087987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.087209    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"
Nov 01 20:52:01.087987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.087222    4872 scope.go:122] "RemoveContainer" containerID="27f46f3be09257d3fd46767d380297de8d578a20bdceea32f048f7104639c2e4"
Nov 01 20:52:01.142002 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.141975    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.144591 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.144568    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkc5\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5\") pod \"e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") " pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.203814 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.203758    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podStartSLOduration=1.203743988 podStartE2EDuration="1.203743988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:01.201863623 +0000 UTC m=+1278.250973442" watchObservedRunningTime="2026-11-01 20:52:01.203743988 +0000 UTC m=+1278.252853807"
Nov 01 20:52:01.243979 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.243940    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:01.363273 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.336727    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"]
Nov 01 20:52:01.363273 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.346835    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ppfz4"]
Nov 01 20:52:01.385222 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.385120    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:01.386482 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.386454    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:52:01.386634 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.386502    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:52:01.634853 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.634823    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"]
Nov 01 20:52:01.635779 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:52:01.635715    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e658443_33c4_46c1_9a7b_6aa7696d23dc.slice/crio-bdd660e2ea4554eab62a9aba59a162591f70bdd64f70d0caa824c87a036892e3 WatchSource:0}: Error finding container bdd660e2ea4554eab62a9aba59a162591f70bdd64f70d0caa824c87a036892e3: Status 404 returned error can't find the container with id bdd660e2ea4554eab62a9aba59a162591f70bdd64f70d0caa824c87a036892e3
Nov 01 20:52:01.638773 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.638640    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:01.638857 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.638825    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:01.638898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:01.638864    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:02.096712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:02.096648    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" event={"ID":"2e658443-33c4-46c1-9a7b-6aa7696d23dc","Type":"ContainerStarted","Data":"9e926c56bbe68abeb39cc92ba7cb1b49c65ba93615d4d09845f1f41268df212f"}
Nov 01 20:52:02.097105 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:02.096732    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" event={"ID":"2e658443-33c4-46c1-9a7b-6aa7696d23dc","Type":"ContainerStarted","Data":"bdd660e2ea4554eab62a9aba59a162591f70bdd64f70d0caa824c87a036892e3"}
Nov 01 20:52:02.216822 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:02.216773    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podStartSLOduration=2.216757656 podStartE2EDuration="2.216757656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:02.213376841 +0000 UTC m=+1279.262486675" watchObservedRunningTime="2026-11-01 20:52:02.216757656 +0000 UTC m=+1279.265867474"
Nov 01 20:52:02.244908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:02.244879    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:02.250328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:02.250301    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:02.389861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:02.389786    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:03.081112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:03.081075    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada7e173-1bca-4229-becb-69e9edc1250f" path="/var/lib/kubelet/pods/ada7e173-1bca-4229-becb-69e9edc1250f/volumes"
Nov 01 20:52:03.102873 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:03.102838    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:03.103341 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:03.103037    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:03.104070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:03.104049    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:52:03.104187 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:03.104121    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:52:11.217383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.217343    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"]
Nov 01 20:52:11.223071 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.223045    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.281610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.281576    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"]
Nov 01 20:52:11.289521 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.289494    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.290301 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.290280    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"]
Nov 01 20:52:11.293031 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.293003    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qgj\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.293754 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.293732    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.293866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.293777    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.293866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.293850    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.293978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.293879    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.293978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.293909    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.294219 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.294202    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.294287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.294235    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.294452 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.294427    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.394708    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"]
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.395709    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.395747    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.395809    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.395837    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.395900    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.395972    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396010 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396016    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qgj\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396066    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396113    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396145    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396169    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpss\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396197    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396234    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396264    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396326    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396357    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396385    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.396565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.396430    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.397480 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.397458    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.397627 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.397474    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.398145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.398118    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.398417 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.398393    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.399190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.399170    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.400177 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.400158    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.403383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.403362    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.456086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.456049    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.456234 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.456144    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qgj\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.497724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.497613    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.497724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.497690    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.497724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.497708    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpss\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.497737    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.497755    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.497885    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.498040    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.498074    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.498091    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498335 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.498266    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.498627    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.498910 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.498872    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.499103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.499084    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.499377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.499355    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.500518 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.500495    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.501405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.501382    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.544782 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.544716    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:11.563088 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.563059    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpss\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.563088 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.563082    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.607577 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.607548    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:11.997986 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:11.997940    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"]
Nov 01 20:52:11.998359 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:52:11.998333    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc63ce5_6af2_4200_a4c0_86ffe970aab9.slice/crio-1b2c90a819c8bfed1c74df782f88d488ddcdfc93170c13b82ff2489f6fb65585 WatchSource:0}: Error finding container 1b2c90a819c8bfed1c74df782f88d488ddcdfc93170c13b82ff2489f6fb65585: Status 404 returned error can't find the container with id 1b2c90a819c8bfed1c74df782f88d488ddcdfc93170c13b82ff2489f6fb65585
Nov 01 20:52:12.001084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.001051    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:12.001180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.001119    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:12.001180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.001157    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:12.050599 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.050574    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"]
Nov 01 20:52:12.065232 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:52:12.065202    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4b9217_3e7b_4f74_83fc_e269ff1a5bff.slice/crio-8e9b450bf098a6af2416d6771ac02de2399ef59da2cb212769bebfb4bed0ba7a WatchSource:0}: Error finding container 8e9b450bf098a6af2416d6771ac02de2399ef59da2cb212769bebfb4bed0ba7a: Status 404 returned error can't find the container with id 8e9b450bf098a6af2416d6771ac02de2399ef59da2cb212769bebfb4bed0ba7a
Nov 01 20:52:12.067899 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.067866    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:12.067999 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.067938    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:12.067999 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.067965    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:12.168863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.168825    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" event={"ID":"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff","Type":"ContainerStarted","Data":"9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9"}
Nov 01 20:52:12.169065 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.168890    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" event={"ID":"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff","Type":"ContainerStarted","Data":"8e9b450bf098a6af2416d6771ac02de2399ef59da2cb212769bebfb4bed0ba7a"}
Nov 01 20:52:12.172842 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.172089    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" event={"ID":"ffc63ce5-6af2-4200-a4c0-86ffe970aab9","Type":"ContainerStarted","Data":"8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb"}
Nov 01 20:52:12.172842 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.172148    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" event={"ID":"ffc63ce5-6af2-4200-a4c0-86ffe970aab9","Type":"ContainerStarted","Data":"1b2c90a819c8bfed1c74df782f88d488ddcdfc93170c13b82ff2489f6fb65585"}
Nov 01 20:52:12.407039 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.406985    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podStartSLOduration=1.406969548 podStartE2EDuration="1.406969548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.403878879 +0000 UTC m=+1289.452988700" watchObservedRunningTime="2026-11-01 20:52:12.406969548 +0000 UTC m=+1289.456079367"
Nov 01 20:52:12.407541 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.407277    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podStartSLOduration=1.407264902 podStartE2EDuration="1.407264902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.28798824 +0000 UTC m=+1289.337098059" watchObservedRunningTime="2026-11-01 20:52:12.407264902 +0000 UTC m=+1289.456374723"
Nov 01 20:52:12.545459 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.545415    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:12.612680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:12.608752    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:13.547332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.547291    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 20:52:13.547958 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.547924    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 20:52:13.562694 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.562632    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"]
Nov 01 20:52:13.571209 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.571179    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:13.578551 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.577431    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:13.578551 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.577575    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgp7v\" (UniqueName: \"kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v\") pod \"payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:13.614190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.612411    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:52:13.614190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.612471    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:52:13.784702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.783407    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0ef08b4\""
Nov 01 20:52:13.792735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.789626    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgp7v\" (UniqueName: \"kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v\") pod \"payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:13.792735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.790068    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:13.820084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:13.814808    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a0ef08b4-dockercfg-fmkd7\""
Nov 01 20:52:14.071743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.056621    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:14.078262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.078227    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"]
Nov 01 20:52:14.095499 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.095461    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgp7v\" (UniqueName: \"kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v\") pod \"payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:14.246409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.246094    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:14.563064 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.563027    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 20:52:14.563724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.563688    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 20:52:14.616681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.614116    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:52:14.616681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.614193    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:52:14.972021 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:14.971991    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"]
Nov 01 20:52:14.978933 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:52:14.978902    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab30e55d_547d_4018_9f6f_cb438608a0cc.slice/crio-ed476f80241fe2d80e0574b364819e3830509e81b9682dc5def67ebdc9b40e86 WatchSource:0}: Error finding container ed476f80241fe2d80e0574b364819e3830509e81b9682dc5def67ebdc9b40e86: Status 404 returned error can't find the container with id ed476f80241fe2d80e0574b364819e3830509e81b9682dc5def67ebdc9b40e86
Nov 01 20:52:15.366706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.366336    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68" event={"ID":"ab30e55d-547d-4018-9f6f-cb438608a0cc","Type":"ContainerStarted","Data":"76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64"}
Nov 01 20:52:15.366706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.366390    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68" event={"ID":"ab30e55d-547d-4018-9f6f-cb438608a0cc","Type":"ContainerStarted","Data":"ed476f80241fe2d80e0574b364819e3830509e81b9682dc5def67ebdc9b40e86"}
Nov 01 20:52:15.366706 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.366416    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:15.492277 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.492209    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68" podStartSLOduration=2.492194858 podStartE2EDuration="2.492194858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:15.491787534 +0000 UTC m=+1292.540897355" watchObservedRunningTime="2026-11-01 20:52:15.492194858 +0000 UTC m=+1292.541304678"
Nov 01 20:52:15.547683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.546070    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 20:52:15.547683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.546139    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 20:52:15.618736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.617862    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:52:15.618736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:15.617920    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:52:16.551830 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:16.551779    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 20:52:16.552071 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:16.551847    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 20:52:16.649757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:16.649709    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:52:16.650233 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:16.649870    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:52:17.017887 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.017849    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"]
Nov 01 20:52:17.023829 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.023795    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.058478 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.058452    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.058614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.058490    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.058614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.058519    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.058760 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.058733    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.058997 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.058970    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhc9f\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.059062 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.059041    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.059103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.059083    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.077976 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.077932    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 20:52:17.078161 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.078003    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 20:52:17.078161 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.078039    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 20:52:17.078161 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.078063    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z57nd\""
Nov 01 20:52:17.154096 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.154059    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"]
Nov 01 20:52:17.160279 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160245    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.160494 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160462    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.161275 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160681    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhc9f\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.161275 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160763    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.161275 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160802    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.161275 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160854    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.161275 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.160897    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.161275 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.161199    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.163604 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.163549    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.163738 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.163596    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.164187 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.164165    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.164888 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.164862    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.249046 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.249007    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhc9f\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.250463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.250437    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmd5t\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.363802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.363698    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:17.548681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.548242    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 20:52:17.548681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.548306    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 20:52:17.644740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.643802    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:52:17.644740 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.643863    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:52:17.850093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.850060    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"]
Nov 01 20:52:17.851236 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.851206    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:17.851397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:17.851381    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:18.396988 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.396945    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" event={"ID":"b479970e-3596-4ee6-bbd2-700dbdcf0d3f","Type":"ContainerStarted","Data":"fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02"}
Nov 01 20:52:18.397190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.397005    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" event={"ID":"b479970e-3596-4ee6-bbd2-700dbdcf0d3f","Type":"ContainerStarted","Data":"91f2fc3b8e98e9e31a4e4eeffc057e301ad9ad2dfd6ee0ed08caa4b7faaf1e5a"}
Nov 01 20:52:18.397316 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.397257    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:18.402435 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.402404    4872 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kmd5t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:52:18.402566 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.402458    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:52:18.519198 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.519133    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" podStartSLOduration=2.5191125100000002 podStartE2EDuration="2.51911251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:18.515485418 +0000 UTC m=+1295.564595236" watchObservedRunningTime="2026-11-01 20:52:18.51911251 +0000 UTC m=+1295.568222329"
Nov 01 20:52:18.546184 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.546147    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 20:52:18.546368 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.546211    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 20:52:18.607964 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.607923    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:52:18.608180 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:18.607994    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:52:19.502676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:19.493871    4872 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kmd5t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:52:19.502676 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:19.493941    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:52:20.042047 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:20.042002    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:20.043382 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:20.043185    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:20.044042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:20.044011    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 20:52:20.187582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:20.187524    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:20.188337 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:20.188304    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:20.202710 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:20.194190    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 20:52:22.485168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:22.485127    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 20:52:23.367409 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:52:23.367368    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:52:25.625741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.624969    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.649341 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.649292    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.668811 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.668757    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.765681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.764758    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.765903 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.765705    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:52:25.769680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.765993    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:52:25.769680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.766248    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.809678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.806777    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.995702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.992783    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:25.995702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:25.995590    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.050225 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.049841    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.255738 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.252820    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.409235 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.407851    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:52:26.737683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.733684    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.737683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.734060    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.737683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.734315    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 20:52:26.737683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.734977    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:52:26.737683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.735036    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:52:26.813278 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.813131    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.839565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.817200    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:52:26.839565 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.818750    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.840373    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.841927    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.842164    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.842331    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.842679    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.842731    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:52:26.863384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.842768    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:52:26.998477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:26.998371    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.042628 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.010776    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 01 20:52:27.047728 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.010809    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:52:27.080985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.080941    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 01 20:52:27.081313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.053867    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.081851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.053812    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:52:27.082072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.082057    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:52:27.082201 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.082190    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:52:27.266151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.266055    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.384712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.382397    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.532739 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.516117    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:52:27.795743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.750763    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.795743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.764770    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.795743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.776197    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 20:52:27.795743 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.789027    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:52:27.820605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.818757    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:27.820605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.818936    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 20:52:27.943678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.853980    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:28.069356 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.854841    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:52:28.070370 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:27.854874    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 20:52:28.081644 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.081600    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 20:52:28.086143 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.049767    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:28.086560 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.068910    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:52:28.112086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.049826    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:28.112668 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.112621    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:52:28.449680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.416726    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:28.882744 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.868789    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:28.907432 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.895760    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded"
Nov 01 20:52:28.958683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:28.952613    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.019059 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.019010    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.059682 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.034975    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:29.195341 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.143700    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.632783 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.625861    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.875318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.875267    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:29.919199 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.877465    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.925788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.888336    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.927304 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.888391    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.943006 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.888415    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.943368 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.888447    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.943608 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.900446    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 20:52:29.943839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:29.907972    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm" podUID="e2403ce9-732c-42f0-b2d4-eb1fb462e1d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.041002 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.040957    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.081681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.050383    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.082209 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.082159    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.187331 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.187222    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 01 20:52:30.373165 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:52:30.366074    4872 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.139s"
Nov 01 20:52:30.399421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.399310    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 20:52:30.440506 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.440419    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 20:52:30.629527 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.629481    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.632151 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.632121    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 20:52:30.652270 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.652238    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm"
Nov 01 20:52:30.865480 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.865385    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 20:52:30.876147 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.876107    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.879372 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.877952    4872 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 01 20:52:30.915454 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.915422    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 20:52:30.947106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.932267    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.947106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.943790    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:52:30.947106 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.944279    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:52:31.017054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.984181    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 20:52:31.017054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.985714    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 20:52:31.017054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.990985    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 20:52:31.017054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.992822    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 20:52:31.017054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:30.998260    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 20:52:31.027694 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:31.026431    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 20:52:31.049487 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:31.048585    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 20:52:53.367340 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:52:53.367297    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:52:55.450217 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.450179    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"]
Nov 01 20:52:55.455869 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.455845    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.509868 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.509818    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\""
Nov 01 20:52:55.556674 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.556621    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.556966 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.556746    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.556966 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.556774    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b393c-7ffa-4964-b8bb-1962b54592e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.556966 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.556891    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.556966 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.556956    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.557214 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.556992    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.557214 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.557016    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjff\" (UniqueName: \"kubernetes.io/projected/a11b393c-7ffa-4964-b8bb-1962b54592e5-kube-api-access-4zjff\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.557214 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.557052    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.557214 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.557110    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.564989 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.564954    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"]
Nov 01 20:52:55.660003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.659639    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.659754    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.659797    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.659821    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjff\" (UniqueName: \"kubernetes.io/projected/a11b393c-7ffa-4964-b8bb-1962b54592e5-kube-api-access-4zjff\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.659857    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660003 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.659916    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.660053    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.660158    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.660184    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b393c-7ffa-4964-b8bb-1962b54592e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.660511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.660376    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.661064 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.661034    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b393c-7ffa-4964-b8bb-1962b54592e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.661399 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.661374    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.661680 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.661641    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.661896 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.661875    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.664294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.664266    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.664527 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.664508    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.717405 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.717333    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b393c-7ffa-4964-b8bb-1962b54592e5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.718451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.718422    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjff\" (UniqueName: \"kubernetes.io/projected/a11b393c-7ffa-4964-b8bb-1962b54592e5-kube-api-access-4zjff\") pod \"maas-default-gateway-openshift-default-58c6585447-qnwxz\" (UID: \"a11b393c-7ffa-4964-b8bb-1962b54592e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:55.770133 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:55.770093    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:56.226588 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:56.226564    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"]
Nov 01 20:52:56.228523 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:52:56.228496    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11b393c_7ffa_4964_b8bb_1962b54592e5.slice/crio-c26b6c4f20d062adb0fb5ddcfbbeb15acf0bd6ee86f6eda4cc655cc7931d9980 WatchSource:0}: Error finding container c26b6c4f20d062adb0fb5ddcfbbeb15acf0bd6ee86f6eda4cc655cc7931d9980: Status 404 returned error can't find the container with id c26b6c4f20d062adb0fb5ddcfbbeb15acf0bd6ee86f6eda4cc655cc7931d9980
Nov 01 20:52:56.232182 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:56.232149    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:56.232272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:56.232226    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:56.232272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:56.232263    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:52:56.903168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:56.903135    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" event={"ID":"a11b393c-7ffa-4964-b8bb-1962b54592e5","Type":"ContainerStarted","Data":"2abdcdb2ae56d299b55c2fec01122b23ca91aafcae91eaf2ce10c952dd5ae952"}
Nov 01 20:52:56.903688 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:56.903212    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" event={"ID":"a11b393c-7ffa-4964-b8bb-1962b54592e5","Type":"ContainerStarted","Data":"c26b6c4f20d062adb0fb5ddcfbbeb15acf0bd6ee86f6eda4cc655cc7931d9980"}
Nov 01 20:52:57.071690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:57.071616    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podStartSLOduration=2.071600604 podStartE2EDuration="2.071600604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:57.019268829 +0000 UTC m=+1334.068378648" watchObservedRunningTime="2026-11-01 20:52:57.071600604 +0000 UTC m=+1334.120710423"
Nov 01 20:52:57.771024 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:57.770989    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:52:57.772908 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:57.772883    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:52:57.773023 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:57.772935    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:52:58.771112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:58.771077    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:52:58.771527 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:58.771136    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:52:59.770987 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:59.770950    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:52:59.771147 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:52:59.771011    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:53:00.771733 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:00.771698    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:53:00.772162 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:00.771748    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:53:01.771639 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:01.771602    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:53:01.771835 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:01.771687    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:53:02.770845 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:02.770805    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:53:02.771060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:02.770881    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:53:03.771223 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:03.771181    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qnwxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 20:53:03.771760 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:03.771251    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz" podUID="a11b393c-7ffa-4964-b8bb-1962b54592e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 20:53:04.991159 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:04.991116    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:53:04.991602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:04.991551    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:53:04.993085 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:04.993065    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qnwxz"
Nov 01 20:53:17.099800 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:17.099764    4872 generic.go:356] "Generic (PLEG): container finished" podID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerID="46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9" exitCode=0
Nov 01 20:53:17.100247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:17.099832    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" event={"ID":"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9","Type":"ContainerDied","Data":"46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"}
Nov 01 20:53:17.100351 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:17.100334    4872 scope.go:122] "RemoveContainer" containerID="46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"
Nov 01 20:53:17.102057 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:17.102028    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:53:17.102122 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:17.102103    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:53:17.102164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:17.102140    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:53:18.112008 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:18.111212    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:53:18.127976 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:18.127942    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" event={"ID":"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9","Type":"ContainerStarted","Data":"a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f"}
Nov 01 20:53:18.384981 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:18.384885    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:53:18.386181 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:18.386154    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:53:18.386336 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:18.386213    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:53:19.384511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:19.384465    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:53:19.384970 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:19.384533    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:53:20.385253 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:20.385213    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:53:20.385787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:20.385288    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:53:21.385352 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:21.385307    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:53:21.385861 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:21.385362    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:53:22.385465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:22.385427    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:53:22.385890 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:22.385483    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:53:23.366750 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:53:23.366698    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:53:23.385234 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:23.385195    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 20:53:23.385442 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:23.385263    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 20:53:24.389484 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:24.389438    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:53:24.389940 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:24.389918    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:53:24.390930 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:24.390909    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:53:53.369022 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:53:53.368979    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:53:53.415889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:53.415857    4872 generic.go:356] "Generic (PLEG): container finished" podID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerID="9e926c56bbe68abeb39cc92ba7cb1b49c65ba93615d4d09845f1f41268df212f" exitCode=0
Nov 01 20:53:53.416075 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:53.415928    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" event={"ID":"2e658443-33c4-46c1-9a7b-6aa7696d23dc","Type":"ContainerDied","Data":"9e926c56bbe68abeb39cc92ba7cb1b49c65ba93615d4d09845f1f41268df212f"}
Nov 01 20:53:53.416413 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:53.416399    4872 scope.go:122] "RemoveContainer" containerID="9e926c56bbe68abeb39cc92ba7cb1b49c65ba93615d4d09845f1f41268df212f"
Nov 01 20:53:53.418161 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:53.418137    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:53:53.418236 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:53.418203    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:53:53.418271 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:53.418237    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:53:54.479929 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:54.479895    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" event={"ID":"2e658443-33c4-46c1-9a7b-6aa7696d23dc","Type":"ContainerStarted","Data":"b9b1f55df5168e8bbacddd547a9d8485b29b13b7ac12cbaffe582aeb57a74afc"}
Nov 01 20:53:55.246504 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:55.246202    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:53:55.247511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:55.246849    4872 patch_prober.go:29] interesting pod/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:53:55.247723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:55.247698    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:53:56.245175 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:56.245134    4872 patch_prober.go:29] interesting pod/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:53:56.245563 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:56.245222    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:53:57.245016 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:57.244979    4872 patch_prober.go:29] interesting pod/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:53:57.245200 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:57.245031    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:53:58.244553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:58.244517    4872 patch_prober.go:29] interesting pod/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:53:58.244757 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:58.244573    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:53:59.244505 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:59.244470    4872 patch_prober.go:29] interesting pod/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:53:59.244995 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:53:59.244529    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:54:00.248788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:00.248759    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:54:00.249269 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:00.249244    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:54:00.249967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:00.249943    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:54:04.653622 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:04.653586    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"]
Nov 01 20:54:09.654867 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.654825    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68" podUID="ab30e55d-547d-4018-9f6f-cb438608a0cc" containerName="payload-pre-processing" containerID="cri-o://76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64" gracePeriod=25
Nov 01 20:54:09.932258 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.932223    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:54:09.983014 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.982987    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v\" (UniqueName: \"kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v\") pod \"ab30e55d-547d-4018-9f6f-cb438608a0cc\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") "
Nov 01 20:54:09.983134 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.983125    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume\") pod \"ab30e55d-547d-4018-9f6f-cb438608a0cc\" (UID: \"ab30e55d-547d-4018-9f6f-cb438608a0cc\") "
Nov 01 20:54:09.983526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.983503    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume" pod "ab30e55d-547d-4018-9f6f-cb438608a0cc" (UID: "ab30e55d-547d-4018-9f6f-cb438608a0cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:09.983835 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.983818    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab30e55d-547d-4018-9f6f-cb438608a0cc-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:09.984982 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:09.984952    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v" pod "ab30e55d-547d-4018-9f6f-cb438608a0cc" (UID: "ab30e55d-547d-4018-9f6f-cb438608a0cc"). InnerVolumeSpecName "kube-api-access-lgp7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:10.085724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.085694    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgp7v\" (UniqueName: \"kubernetes.io/projected/ab30e55d-547d-4018-9f6f-cb438608a0cc-kube-api-access-lgp7v\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:10.620274 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.620236    4872 generic.go:356] "Generic (PLEG): container finished" podID="ab30e55d-547d-4018-9f6f-cb438608a0cc" containerID="76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64" exitCode=0
Nov 01 20:54:10.620451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.620287    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"
Nov 01 20:54:10.620451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.620317    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68" event={"ID":"ab30e55d-547d-4018-9f6f-cb438608a0cc","Type":"ContainerDied","Data":"76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64"}
Nov 01 20:54:10.620451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.620373    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68" event={"ID":"ab30e55d-547d-4018-9f6f-cb438608a0cc","Type":"ContainerDied","Data":"ed476f80241fe2d80e0574b364819e3830509e81b9682dc5def67ebdc9b40e86"}
Nov 01 20:54:10.620451 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.620396    4872 scope.go:122] "RemoveContainer" containerID="76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64"
Nov 01 20:54:10.631866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.631851    4872 scope.go:122] "RemoveContainer" containerID="76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64"
Nov 01 20:54:10.632119 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:54:10.632102    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64\": container with ID starting with 76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64 not found: ID does not exist" containerID="76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64"
Nov 01 20:54:10.632176 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.632126    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64"} err="failed to get container status \"76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64\": rpc error: code = NotFound desc = could not find container \"76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64\": container with ID starting with 76e82f69db63ee48cc4b083743e9f0054266637c8b79a01350e9a19cf8d8da64 not found: ID does not exist"
Nov 01 20:54:10.786911 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.786886    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"]
Nov 01 20:54:10.837516 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:10.837494    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0ef08b4-579f9b795-ffw68"]
Nov 01 20:54:11.080101 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:11.080074    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab30e55d-547d-4018-9f6f-cb438608a0cc" path="/var/lib/kubelet/pods/ab30e55d-547d-4018-9f6f-cb438608a0cc/volumes"
Nov 01 20:54:23.364562 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:54:23.364524    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:54:38.591601 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:38.591562    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"]
Nov 01 20:54:38.592049 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:38.591823    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy" containerID="cri-o://b9b1f55df5168e8bbacddd547a9d8485b29b13b7ac12cbaffe582aeb57a74afc" gracePeriod=30
Nov 01 20:54:38.810228 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:38.810187    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"]
Nov 01 20:54:38.810418 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:38.810388    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" containerID="cri-o://a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f" gracePeriod=30
Nov 01 20:54:39.392247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:39.392206    4872 patch_prober.go:29] interesting pod/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:54:39.392411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:39.392275    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:54:41.272364 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.272329    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"]
Nov 01 20:54:41.273037 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.273021    4872 cpu_manager.go:414] "containerMap: removing container" podUID="ab30e55d-547d-4018-9f6f-cb438608a0cc" containerName="payload-pre-processing"
Nov 01 20:54:41.273081 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.273038    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab30e55d-547d-4018-9f6f-cb438608a0cc" containerName="payload-pre-processing"
Nov 01 20:54:41.273164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.273154    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab30e55d-547d-4018-9f6f-cb438608a0cc" containerName="payload-pre-processing"
Nov 01 20:54:41.276772 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.276754    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.378686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.378628    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"]
Nov 01 20:54:41.415605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.415571    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.415605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.415602    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.415845 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.415724    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.415845 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.415837    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.415965 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.415871    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.416084 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.416055    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.416179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.416164    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.416242 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.416225    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.416315 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.416299    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gfg\" (UniqueName: \"kubernetes.io/projected/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-kube-api-access-g9gfg\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.517863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.517821    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.517925    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gfg\" (UniqueName: \"kubernetes.io/projected/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-kube-api-access-g9gfg\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.517959    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.517981    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518023    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518264 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518131    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518264 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518187    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518358 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518284    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518358 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518308    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518358 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518326    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518514 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518394    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518534    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518732 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518710    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.518839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.518814    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.520408 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.520381    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.520625 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.520609    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.576271 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.576209    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gfg\" (UniqueName: \"kubernetes.io/projected/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-kube-api-access-g9gfg\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.576773 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.576756    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4102344f-e1f1-4c5a-a1b9-a57ba4e86b30-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-nd97z\" (UID: \"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:41.590768 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:41.590748    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:42.020965 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:42.020935    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"]
Nov 01 20:54:42.022849 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:54:42.022821    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4102344f_e1f1_4c5a_a1b9_a57ba4e86b30.slice/crio-582cff890cabc52a73917ba34a5f67c74613c0264303a08669b256e07d9a1e11 WatchSource:0}: Error finding container 582cff890cabc52a73917ba34a5f67c74613c0264303a08669b256e07d9a1e11: Status 404 returned error can't find the container with id 582cff890cabc52a73917ba34a5f67c74613c0264303a08669b256e07d9a1e11
Nov 01 20:54:42.025605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:42.025571    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:54:42.025736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:42.025646    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:54:42.025736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:42.025728    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:54:42.880625 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:42.880586    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" event={"ID":"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30","Type":"ContainerStarted","Data":"bb01f0b21af391cd935c20cba9b73cc47b0666509c4079250f73db943bf781f0"}
Nov 01 20:54:42.881063 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:42.880648    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" event={"ID":"4102344f-e1f1-4c5a-a1b9-a57ba4e86b30","Type":"ContainerStarted","Data":"582cff890cabc52a73917ba34a5f67c74613c0264303a08669b256e07d9a1e11"}
Nov 01 20:54:43.065076 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.065029    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podStartSLOduration=2.065016863 podStartE2EDuration="2.065016863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:43.048953406 +0000 UTC m=+1440.098063227" watchObservedRunningTime="2026-11-01 20:54:43.065016863 +0000 UTC m=+1440.114126722"
Nov 01 20:54:43.591272 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.591188    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:43.592468 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.592443    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 20:54:43.592523 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.592492    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 20:54:43.889244 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.889215    4872 generic.go:356] "Generic (PLEG): container finished" podID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerID="b9b1f55df5168e8bbacddd547a9d8485b29b13b7ac12cbaffe582aeb57a74afc" exitCode=0
Nov 01 20:54:43.889578 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.889291    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" event={"ID":"2e658443-33c4-46c1-9a7b-6aa7696d23dc","Type":"ContainerDied","Data":"b9b1f55df5168e8bbacddd547a9d8485b29b13b7ac12cbaffe582aeb57a74afc"}
Nov 01 20:54:43.889578 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:43.889359    4872 scope.go:122] "RemoveContainer" containerID="9e926c56bbe68abeb39cc92ba7cb1b49c65ba93615d4d09845f1f41268df212f"
Nov 01 20:54:44.174433 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.174114    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:54:44.362555 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.362352    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.363988 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.363963    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364027    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364076    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364196 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364113    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364230 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364196    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364230 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364218    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364235    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.364288 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.364258    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo\") pod \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\" (UID: \"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9\") "
Nov 01 20:54:44.365642 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.365611    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.365992 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.365895    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.366094 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.366027    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.366768 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.366744    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:44.366885 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.366793    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.368291 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.368261    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:54:44.368396 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.368375    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "kube-api-access-zxdf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:44.368443 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.368380    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.370090 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.370070    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token" pod "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" (UID: "26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:44.383171 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.382569    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:54:44.467481 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467450    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467505    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467533    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467596    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467683 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467651    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467726    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467746    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.467884 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467841    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.468029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.467904    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo\") pod \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\" (UID: \"2e658443-33c4-46c1-9a7b-6aa7696d23dc\") "
Nov 01 20:54:44.468479 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468455    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468485    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468501    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468496    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468515    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468503    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468530    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468543    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468556    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.468569 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468569    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.469013 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468582    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxdf2\" (UniqueName: \"kubernetes.io/projected/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9-kube-api-access-zxdf2\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.469013 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468743    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.469013 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.468755    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.469319 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.469298    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:44.470577 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.470556    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "kube-api-access-brkc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:44.470759 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.470723    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:44.470971 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.470953    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:54:44.471022 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.470966    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy" pod "2e658443-33c4-46c1-9a7b-6aa7696d23dc" (UID: "2e658443-33c4-46c1-9a7b-6aa7696d23dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:54:44.570643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570613    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570643 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570640    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570650    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570677    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570688    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570696    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570704    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570713    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkc5\" (UniqueName: \"kubernetes.io/projected/2e658443-33c4-46c1-9a7b-6aa7696d23dc-kube-api-access-brkc5\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.570839 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.570722    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e658443-33c4-46c1-9a7b-6aa7696d23dc-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:54:44.592128 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.592063    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 20:54:44.592128 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.592114    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 20:54:44.922977 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.922947    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"
Nov 01 20:54:44.923362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.923153    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv" event={"ID":"2e658443-33c4-46c1-9a7b-6aa7696d23dc","Type":"ContainerDied","Data":"bdd660e2ea4554eab62a9aba59a162591f70bdd64f70d0caa824c87a036892e3"}
Nov 01 20:54:44.923362 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.923203    4872 scope.go:122] "RemoveContainer" containerID="b9b1f55df5168e8bbacddd547a9d8485b29b13b7ac12cbaffe582aeb57a74afc"
Nov 01 20:54:44.925815 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.925790    4872 generic.go:356] "Generic (PLEG): container finished" podID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerID="a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f" exitCode=0
Nov 01 20:54:44.926126 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.926107    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"
Nov 01 20:54:44.926207 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.926173    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" event={"ID":"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9","Type":"ContainerDied","Data":"a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f"}
Nov 01 20:54:44.926259 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.926245    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq" event={"ID":"26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9","Type":"ContainerDied","Data":"e635eda13c748e1126aabe8f196f86f6edd9e7698d8a7a3052e1d15e0dff6943"}
Nov 01 20:54:44.941342 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.941321    4872 scope.go:122] "RemoveContainer" containerID="a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f"
Nov 01 20:54:44.955810 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.955470    4872 scope.go:122] "RemoveContainer" containerID="46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"
Nov 01 20:54:44.976860 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.975399    4872 scope.go:122] "RemoveContainer" containerID="a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f"
Nov 01 20:54:44.976860 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:54:44.975741    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f\": container with ID starting with a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f not found: ID does not exist" containerID="a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f"
Nov 01 20:54:44.976860 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.975767    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f"} err="failed to get container status \"a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f\": rpc error: code = NotFound desc = could not find container \"a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f\": container with ID starting with a00395401464c9bf1dfb9be95c46e2daabb733f3a2389d8be332564351a45d3f not found: ID does not exist"
Nov 01 20:54:44.976860 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.975784    4872 scope.go:122] "RemoveContainer" containerID="46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"
Nov 01 20:54:44.976860 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:54:44.976011    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9\": container with ID starting with 46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9 not found: ID does not exist" containerID="46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"
Nov 01 20:54:44.976860 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:44.976025    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9"} err="failed to get container status \"46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9\": rpc error: code = NotFound desc = could not find container \"46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9\": container with ID starting with 46fafd78232a0e36935e7a67cde6ab3869220bbc0212484a580da9a382795cc9 not found: ID does not exist"
Nov 01 20:54:45.130291 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:45.130258    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"]
Nov 01 20:54:45.137965 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:45.137939    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0ef08b4-openshift-default-58c89679c-qf9jq"]
Nov 01 20:54:45.240178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:45.236732    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"]
Nov 01 20:54:45.250712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:45.248856    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d0d57533-openshift-default-bb8564d89-mmmrv"]
Nov 01 20:54:45.591735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:45.591244    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 20:54:45.591735 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:45.591327    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 20:54:46.591235 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:46.591190    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 20:54:46.591608 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:46.591265    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 20:54:47.081147 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:47.081112    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" path="/var/lib/kubelet/pods/26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9/volumes"
Nov 01 20:54:47.081941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:47.081922    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" path="/var/lib/kubelet/pods/2e658443-33c4-46c1-9a7b-6aa7696d23dc/volumes"
Nov 01 20:54:47.591543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:47.591509    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 20:54:47.591958 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:47.591565    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 20:54:48.591273 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:48.591238    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 01 20:54:48.591446 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:48.591296    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 01 20:54:50.520136 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:50.520096    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:50.520690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:50.520666    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:50.521295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:50.521276    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z"
Nov 01 20:54:53.367940 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:54:53.366459    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:54:56.393000 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.392962    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"]
Nov 01 20:54:56.393866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.393845    4872 cpu_manager.go:414] "containerMap: removing container" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy"
Nov 01 20:54:56.393866 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.393867    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy"
Nov 01 20:54:56.394044 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.393890    4872 cpu_manager.go:414] "containerMap: removing container" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy"
Nov 01 20:54:56.394044 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.393898    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy"
Nov 01 20:54:56.394044 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.393929    4872 cpu_manager.go:414] "containerMap: removing container" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy"
Nov 01 20:54:56.394044 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.393936    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy"
Nov 01 20:54:56.394231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.394148    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy"
Nov 01 20:54:56.394231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.394165    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy"
Nov 01 20:54:56.394231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.394177    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e658443-33c4-46c1-9a7b-6aa7696d23dc" containerName="istio-proxy"
Nov 01 20:54:56.394231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.394196    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy"
Nov 01 20:54:56.399234 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.399214    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.509813 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.509779    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"]
Nov 01 20:54:56.510781 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.510755    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.510883 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.510797    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.510937 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.510920    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.511081 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.511065    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.511118 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.511108    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.511197 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.511183    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.511371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.511349    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr9d\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.511443 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.511427    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.511485 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.511456    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612325 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612296    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612337    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612364    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612477 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612444    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr9d\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612491    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612505    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612718 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612694    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612782 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612744    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612835 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612823    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.612889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612833    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.613022 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.612999    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.613168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.613136    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.613239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.613171    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.613239 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.613188    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.614773 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.614744    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.614976 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.614961    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.669669 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.669641    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.669768 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.669755    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr9d\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d\") pod \"maas-default-gateway-openshift-default-58c6585447-82z67\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:56.712315 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:56.712290    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:57.147276 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:57.147246    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"]
Nov 01 20:54:57.147747 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:54:57.147714    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2444cc_8b9f_42de_8632_d903c7923a2c.slice/crio-03d006cb4874ef61b1f266ceb8e01177db2d5a4c3bc15acb526b24c0a64822dd WatchSource:0}: Error finding container 03d006cb4874ef61b1f266ceb8e01177db2d5a4c3bc15acb526b24c0a64822dd: Status 404 returned error can't find the container with id 03d006cb4874ef61b1f266ceb8e01177db2d5a4c3bc15acb526b24c0a64822dd
Nov 01 20:54:57.150002 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:57.149975    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:54:57.150088 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:57.150031    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:54:57.150088 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:57.150058    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:54:58.075598 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:58.075560    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" event={"ID":"5f2444cc-8b9f-42de-8632-d903c7923a2c","Type":"ContainerStarted","Data":"ab3f563bfeecac2cd0e929fe46c2b6d4e9b2752137045076b36bba1ce90b7649"}
Nov 01 20:54:58.076347 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:58.076312    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" event={"ID":"5f2444cc-8b9f-42de-8632-d903c7923a2c","Type":"ContainerStarted","Data":"03d006cb4874ef61b1f266ceb8e01177db2d5a4c3bc15acb526b24c0a64822dd"}
Nov 01 20:54:58.314678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:58.314071    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podStartSLOduration=2.314049435 podStartE2EDuration="2.314049435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:58.292702024 +0000 UTC m=+1455.341811841" watchObservedRunningTime="2026-11-01 20:54:58.314049435 +0000 UTC m=+1455.363159256"
Nov 01 20:54:58.714950 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:58.714913    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:54:58.715300 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:58.715277    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:54:58.715484 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:58.715458    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:54:59.722112 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:59.722077    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:54:59.722719 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:54:59.722686    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:55:00.716679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:00.716169    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:55:00.716679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:00.716290    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:55:01.719146 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:01.719112    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:55:01.719745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:01.719715    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:55:02.716678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:02.716081    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:55:02.716678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:02.716146    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:55:03.713289 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:03.713254    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:55:03.713794 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:03.713325    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:55:05.468542 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:05.468496    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:55:05.469268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:05.469170    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:55:05.470179 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:05.470155    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 20:55:07.819210 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.819179    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"]
Nov 01 20:55:07.820093 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.820072    4872 cpu_manager.go:414] "containerMap: removing container" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy"
Nov 01 20:55:07.820153 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.820096    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ff28c9-a1d1-4ab8-9a2d-56be971bd8d9" containerName="istio-proxy"
Nov 01 20:55:07.824578 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.824557    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:07.901978 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.901938    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"]
Nov 01 20:55:07.925809 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.925769    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a5ab4417\""
Nov 01 20:55:07.926018 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.925909    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a5ab4417-dockercfg-pl5wc\""
Nov 01 20:55:07.962133 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.962095    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:07.962818 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.962781    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrwp\" (UniqueName: \"kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp\") pod \"payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:07.976481 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.976458    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"]
Nov 01 20:55:07.980837 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:07.980820    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.038087 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.038057    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"]
Nov 01 20:55:08.065137 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.065102    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:08.065355 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.065190    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrwp\" (UniqueName: \"kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp\") pod \"payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:08.065355 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.065240    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume\") pod \"payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") " pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.065492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.065457    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58spb\" (UniqueName: \"kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb\") pod \"payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") " pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.066292 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.066260    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:08.130556 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.130478    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrwp\" (UniqueName: \"kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp\") pod \"payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:08.140621 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.140592    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:08.189677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.173830    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58spb\" (UniqueName: \"kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb\") pod \"payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") " pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.189677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.174123    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume\") pod \"payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") " pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.189677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.175035    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume\") pod \"payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") " pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.288802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.288767    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58spb\" (UniqueName: \"kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb\") pod \"payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") " pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.299525 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.299480    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:08.635852 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.635826    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"]
Nov 01 20:55:08.636879 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:08.636854    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e8391f_cf8b_4f90_8e0b_c130634ee2c3.slice/crio-31a9997c23406dc30d7fbb33a54c0dfac356b103ec060242fa1668a9e0f62ee0 WatchSource:0}: Error finding container 31a9997c23406dc30d7fbb33a54c0dfac356b103ec060242fa1668a9e0f62ee0: Status 404 returned error can't find the container with id 31a9997c23406dc30d7fbb33a54c0dfac356b103ec060242fa1668a9e0f62ee0
Nov 01 20:55:08.737881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:08.737852    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"]
Nov 01 20:55:08.738234 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:08.738213    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda74b6a9_6e87_4818_8cc9_b64e9da8403b.slice/crio-0b501dc5a3d98ac814101855941b9dd98f3e7bd40d76cc356b5d12982c0afa14 WatchSource:0}: Error finding container 0b501dc5a3d98ac814101855941b9dd98f3e7bd40d76cc356b5d12982c0afa14: Status 404 returned error can't find the container with id 0b501dc5a3d98ac814101855941b9dd98f3e7bd40d76cc356b5d12982c0afa14
Nov 01 20:55:09.298928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.298891    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76" event={"ID":"da74b6a9-6e87-4818-8cc9-b64e9da8403b","Type":"ContainerStarted","Data":"5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d"}
Nov 01 20:55:09.299390 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.298951    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76" event={"ID":"da74b6a9-6e87-4818-8cc9-b64e9da8403b","Type":"ContainerStarted","Data":"0b501dc5a3d98ac814101855941b9dd98f3e7bd40d76cc356b5d12982c0afa14"}
Nov 01 20:55:09.299390 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.298979    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:09.300610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.300586    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2" event={"ID":"83e8391f-cf8b-4f90-8e0b-c130634ee2c3","Type":"ContainerStarted","Data":"043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31"}
Nov 01 20:55:09.300753 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.300627    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2" event={"ID":"83e8391f-cf8b-4f90-8e0b-c130634ee2c3","Type":"ContainerStarted","Data":"31a9997c23406dc30d7fbb33a54c0dfac356b103ec060242fa1668a9e0f62ee0"}
Nov 01 20:55:09.300753 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.300670    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:09.416758 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.416712    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76" podStartSLOduration=2.416699407 podStartE2EDuration="2.416699407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:09.412496605 +0000 UTC m=+1466.461606419" watchObservedRunningTime="2026-11-01 20:55:09.416699407 +0000 UTC m=+1466.465809224"
Nov 01 20:55:09.528348 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:09.528295    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2" podStartSLOduration=2.528282258 podStartE2EDuration="2.528282258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:09.525945856 +0000 UTC m=+1466.575055673" watchObservedRunningTime="2026-11-01 20:55:09.528282258 +0000 UTC m=+1466.577392075"
Nov 01 20:55:12.693546 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.693510    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"]
Nov 01 20:55:12.698354 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.698334    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.746959 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.746924    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.747213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.747188    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gndv\" (UniqueName: \"kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.752867 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.752839    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"]
Nov 01 20:55:12.798946 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.798886    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-11a069b1\""
Nov 01 20:55:12.799135 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.798965    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-11a069b1-dockercfg-64n9c\""
Nov 01 20:55:12.847563 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.847527    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"]
Nov 01 20:55:12.848550 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.848520    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.848967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.848877    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gndv\" (UniqueName: \"kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.850265 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.850149    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.853243 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.853092    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:12.918916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.914797    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"]
Nov 01 20:55:12.918916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.918763    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gndv\" (UniqueName: \"kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:12.951071 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.950853    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:12.951281 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:12.951087    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bwf\" (UniqueName: \"kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:13.015369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.015330    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:13.052287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.052257    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:13.052495 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.052479    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bwf\" (UniqueName: \"kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:13.052986 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.052966    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:13.126689 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.122265    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bwf\" (UniqueName: \"kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:13.190049 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.190016    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:13.402716 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.402688    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"]
Nov 01 20:55:13.403320 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:13.403297    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b767ec_a136_4b44_b740_79f223592863.slice/crio-0fe61ad9db5cf471fab8e63d7e1e72338e33cb194f7e30b37435bbe8ddbf3c00 WatchSource:0}: Error finding container 0fe61ad9db5cf471fab8e63d7e1e72338e33cb194f7e30b37435bbe8ddbf3c00: Status 404 returned error can't find the container with id 0fe61ad9db5cf471fab8e63d7e1e72338e33cb194f7e30b37435bbe8ddbf3c00
Nov 01 20:55:13.608083 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:13.608059    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"]
Nov 01 20:55:13.609527 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:13.609501    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaee4916_972b_4f58_a7b5_9027dd759157.slice/crio-c65e809f83116ff783d3ac40f625eca945d14086a75dd23c28c566cd97d8aea8 WatchSource:0}: Error finding container c65e809f83116ff783d3ac40f625eca945d14086a75dd23c28c566cd97d8aea8: Status 404 returned error can't find the container with id c65e809f83116ff783d3ac40f625eca945d14086a75dd23c28c566cd97d8aea8
Nov 01 20:55:14.346739 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.346696    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" event={"ID":"28b767ec-a136-4b44-b740-79f223592863","Type":"ContainerStarted","Data":"ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b"}
Nov 01 20:55:14.347199 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.346774    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:14.347199 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.346799    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" event={"ID":"28b767ec-a136-4b44-b740-79f223592863","Type":"ContainerStarted","Data":"0fe61ad9db5cf471fab8e63d7e1e72338e33cb194f7e30b37435bbe8ddbf3c00"}
Nov 01 20:55:14.348610 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.348585    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" event={"ID":"daee4916-972b-4f58-a7b5-9027dd759157","Type":"ContainerStarted","Data":"14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a"}
Nov 01 20:55:14.348730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.348625    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" event={"ID":"daee4916-972b-4f58-a7b5-9027dd759157","Type":"ContainerStarted","Data":"c65e809f83116ff783d3ac40f625eca945d14086a75dd23c28c566cd97d8aea8"}
Nov 01 20:55:14.348730 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.348693    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:14.464221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.464169    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" podStartSLOduration=2.464154497 podStartE2EDuration="2.464154497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:14.460412117 +0000 UTC m=+1471.509521937" watchObservedRunningTime="2026-11-01 20:55:14.464154497 +0000 UTC m=+1471.513264314"
Nov 01 20:55:14.584828 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:14.584772    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" podStartSLOduration=2.584754708 podStartE2EDuration="2.584754708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:14.577523645 +0000 UTC m=+1471.626633464" watchObservedRunningTime="2026-11-01 20:55:14.584754708 +0000 UTC m=+1471.633864526"
Nov 01 20:55:20.318856 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:20.318826    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:55:20.319241 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:20.318892    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:55:23.203720 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.203687    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"]
Nov 01 20:55:23.208384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.208360    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.262055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.262023    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8ddfb06d-serving-cert\""
Nov 01 20:55:23.305999 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.305960    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.306145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.306021    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.306208 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.306119    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6kfd\" (UniqueName: \"kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.306278 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.306254    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.316840 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.316814    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"]
Nov 01 20:55:23.374035 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:23.374003    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:55:23.407500 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.407474    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.407677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.407522    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.407677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.407554    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6kfd\" (UniqueName: \"kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.407677 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.407627    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.407858 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:23.407710    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-serving-cert: secret "maas-api-e2e-migrate-8ddfb06d-serving-cert" not found
Nov 01 20:55:23.407858 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:23.407796    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls podName:42e36f34-a9f1-4ac4-929a-c4c8c09d55b1 nodeName:}" failed. No retries permitted until 2026-11-01 20:55:23.907773027 +0000 UTC m=+1480.956882838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls") pod "maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" (UID: "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1") : secret "maas-api-e2e-migrate-8ddfb06d-serving-cert" not found
Nov 01 20:55:23.408341 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.408322    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.410088 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.410069    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.467167 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.467080    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6kfd\" (UniqueName: \"kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.917955 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.917924    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:23.920317 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:23.920297    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\") pod \"maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:24.123343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:24.123314    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:24.500313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:24.500285    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"]
Nov 01 20:55:24.500969 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:24.500947    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e36f34_a9f1_4ac4_929a_c4c8c09d55b1.slice/crio-ba3b5b90026053c499958612cbd449862d5edeac9500942dbfef410f4fc22ddb WatchSource:0}: Error finding container ba3b5b90026053c499958612cbd449862d5edeac9500942dbfef410f4fc22ddb: Status 404 returned error can't find the container with id ba3b5b90026053c499958612cbd449862d5edeac9500942dbfef410f4fc22ddb
Nov 01 20:55:24.502298 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:24.502279    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:55:25.359383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:25.359306    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 20:55:25.359526 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:25.359460    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 20:55:25.429449 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:25.429423    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" event={"ID":"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1","Type":"ContainerStarted","Data":"ba3b5b90026053c499958612cbd449862d5edeac9500942dbfef410f4fc22ddb"}
Nov 01 20:55:26.444871 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:26.444835    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" event={"ID":"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1","Type":"ContainerStarted","Data":"68ff60448bdfd30f4877b6f9933a005c5a32152bb5ad7a9bc37d0cc0a506b898"}
Nov 01 20:55:26.445374 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:26.444971    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:26.562032 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:26.561984    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" podStartSLOduration=2.663543244 podStartE2EDuration="3.561971013s" totalImagesPullingTime="898.427769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:26.560477727 +0000 UTC m=+1483.609587545" watchObservedRunningTime="2026-11-01 20:55:26.561971013 +0000 UTC m=+1483.611080830"
Nov 01 20:55:27.761756 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.761725    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"]
Nov 01 20:55:27.765543 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.765524    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.822166 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.822144    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"]
Nov 01 20:55:27.868282 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.868256    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-74ab02fe-serving-cert\""
Nov 01 20:55:27.869263 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.869229    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpxg\" (UniqueName: \"kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.869373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.869348    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.869553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.869513    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.869672 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.869610    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.971101 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.971072    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.971238 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.971165    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.971238 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.971233    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.971379 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:27.971362    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-serving-cert: secret "maas-api-e2e-ait-74ab02fe-serving-cert" not found
Nov 01 20:55:27.971430 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.971413    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpxg\" (UniqueName: \"kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.971481 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:27.971461    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls podName:de25f9c8-2083-497b-9547-d6688fb5e7e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:55:28.471431533 +0000 UTC m=+1485.520541340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls") pod "maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" (UID: "de25f9c8-2083-497b-9547-d6688fb5e7e7") : secret "maas-api-e2e-ait-74ab02fe-serving-cert" not found
Nov 01 20:55:27.971870 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.971851    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:27.973480 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:27.973458    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:28.040176 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:28.040119    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpxg\" (UniqueName: \"kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:28.496592 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:28.496557    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:28.499113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:28.499088    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\") pod \"maas-api-e2e-ait-74ab02fe-777d6bd454-j899l\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:28.680554 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:28.680524    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:29.063880 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:29.063850    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"]
Nov 01 20:55:29.064292 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:29.064256    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde25f9c8_2083_497b_9547_d6688fb5e7e7.slice/crio-8e81aa3b56aa326fcde4a5508d6e55b568d015836d8b914e9f720a96aaba6343 WatchSource:0}: Error finding container 8e81aa3b56aa326fcde4a5508d6e55b568d015836d8b914e9f720a96aaba6343: Status 404 returned error can't find the container with id 8e81aa3b56aa326fcde4a5508d6e55b568d015836d8b914e9f720a96aaba6343
Nov 01 20:55:29.465900 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:29.465866    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" event={"ID":"de25f9c8-2083-497b-9547-d6688fb5e7e7","Type":"ContainerStarted","Data":"8e81aa3b56aa326fcde4a5508d6e55b568d015836d8b914e9f720a96aaba6343"}
Nov 01 20:55:30.475373 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:30.475335    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" event={"ID":"de25f9c8-2083-497b-9547-d6688fb5e7e7","Type":"ContainerStarted","Data":"ece9eafdf5a98f91c4377618e0fa746552631aaf3ec5b24ae90ebde61e787274"}
Nov 01 20:55:30.475786 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:30.475403    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:30.595328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:30.595268    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" podStartSLOduration=2.8253130520000003 podStartE2EDuration="3.595252854s" totalImagesPullingTime="769.939802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:30.590990466 +0000 UTC m=+1487.640100283" watchObservedRunningTime="2026-11-01 20:55:30.595252854 +0000 UTC m=+1487.644362673"
Nov 01 20:55:32.457816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:32.457785    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:55:36.487952 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:36.487924    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:55:43.395047 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:43.395019    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:55:43.395463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:43.395375    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 20:55:43.402017 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:43.401993    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:55:43.402913 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:43.402892    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 20:55:51.971703 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:51.971640    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"]
Nov 01 20:55:51.978611 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:51.978582    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.033800 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.033761    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"]
Nov 01 20:55:52.079067 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.079038    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a5ab4417-serving-cert\""
Nov 01 20:55:52.116262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.116230    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"]
Nov 01 20:55:52.125153 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.125119    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.138463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.138437    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wdc\" (UniqueName: \"kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.138741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.138720    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.139206 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.139186    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.139378 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.139362    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.180973 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.180950    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-dce296-openshift-default-dockercfg-8rb4f\""
Nov 01 20:55:52.221317 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.221290    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"]
Nov 01 20:55:52.241592 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.241515    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.241592 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.241576    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.241843 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.241639    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.241843 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.241724    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.241843 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:52.241758    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-serving-cert: secret "maas-api-e2e-mt-a5ab4417-serving-cert" not found
Nov 01 20:55:52.241843 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:52.241815    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls podName:db52d2a0-5f81-4a2b-8172-4de59b3e7eab nodeName:}" failed. No retries permitted until 2026-11-01 20:55:52.741794575 +0000 UTC m=+1509.790904384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls") pod "maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" (UID: "db52d2a0-5f81-4a2b-8172-4de59b3e7eab") : secret "maas-api-e2e-mt-a5ab4417-serving-cert" not found
Nov 01 20:55:52.242060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.241850    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.242060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.241950    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.242060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242014    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.242213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242133    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wdc\" (UniqueName: \"kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.242213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242193    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.242318 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242234    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.242369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242313    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.242457 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242435    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxk6n\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.242515 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.242470    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.243213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.243189    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.244573 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.244554    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.306796 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.306765    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wdc\" (UniqueName: \"kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.344340 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344299    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxk6n\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344528 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344352    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344528 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344470    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344642 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344559    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344642 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344629    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344740    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344797    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344923 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344880    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.344991 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.344919    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.345072 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.345048    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.345423 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.345338    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.345669 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.345636    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.345816 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.345790    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.346011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.345980    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.348605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.348584    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.349429 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.349409    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.402065 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.402037    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.402266 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.402243    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxk6n\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n\") pod \"e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") " pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.445162 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.445121    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:52.754550 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.754469    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.757013 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.756987    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\") pod \"maas-api-e2e-mt-a5ab4417-6494d5595f-w276l\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:52.883913 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.883881    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"]
Nov 01 20:55:52.888895 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.887958    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:55:52.888895 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.888041    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:55:52.888895 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.888085    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:55:52.895378 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:52.895355    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:53.297502 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:55:53.297465    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb52d2a0_5f81_4a2b_8172_4de59b3e7eab.slice/crio-229820a368703c4921c2905d9beaf7b41436e7c7a4172aa37ebd7788ae8dbe1d WatchSource:0}: Error finding container 229820a368703c4921c2905d9beaf7b41436e7c7a4172aa37ebd7788ae8dbe1d: Status 404 returned error can't find the container with id 229820a368703c4921c2905d9beaf7b41436e7c7a4172aa37ebd7788ae8dbe1d
Nov 01 20:55:53.298190 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:53.298158    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"]
Nov 01 20:55:53.367595 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:55:53.367562    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:55:53.639165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:53.639080    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" event={"ID":"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f","Type":"ContainerStarted","Data":"1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"}
Nov 01 20:55:53.639165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:53.639133    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" event={"ID":"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f","Type":"ContainerStarted","Data":"3525ffc64c08eb1ef89738d7a2b93342396c7105955ab2958c2c99e35dde0ac9"}
Nov 01 20:55:53.640493 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:53.640458    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" event={"ID":"db52d2a0-5f81-4a2b-8172-4de59b3e7eab","Type":"ContainerStarted","Data":"229820a368703c4921c2905d9beaf7b41436e7c7a4172aa37ebd7788ae8dbe1d"}
Nov 01 20:55:53.760802 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:53.760757    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podStartSLOduration=1.7607452989999999 podStartE2EDuration="1.760745299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:53.756921701 +0000 UTC m=+1510.806031523" watchObservedRunningTime="2026-11-01 20:55:53.760745299 +0000 UTC m=+1510.809855116"
Nov 01 20:55:54.446229 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.446188    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:54.451997 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.451970    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:54.648830 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.648795    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" event={"ID":"db52d2a0-5f81-4a2b-8172-4de59b3e7eab","Type":"ContainerStarted","Data":"ab0ccdf77136be8a78431e1fa0b97858042fe4f61bb1494ff0c69cf0d6811d8a"}
Nov 01 20:55:54.648997 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.648871    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:55:54.649332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.649311    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:54.650306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.650287    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:55:54.765828 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:55:54.765730    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" podStartSLOduration=2.9435217160000002 podStartE2EDuration="3.765710832s" totalImagesPullingTime="822.189116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:54.76498238 +0000 UTC m=+1511.814092198" watchObservedRunningTime="2026-11-01 20:55:54.765710832 +0000 UTC m=+1511.814820651"
Nov 01 20:56:00.672523 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:00.672492    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:56:12.483939 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:12.483907    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"]
Nov 01 20:56:12.484325 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:12.484089    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" containerID="cri-o://db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243" gracePeriod=30
Nov 01 20:56:16.196557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:16.196524    4872 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:56:16.196973 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:16.196584    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:56:17.802917 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.802891    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:56:17.864044 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864015    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864168 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864083    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864230 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864163    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864230 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864220    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864335 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864301    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864335 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864282    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:56:17.864438 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864342    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:56:17.864438 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864363    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864537 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864509    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864558    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864590 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864570    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:56:17.864728 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864596    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token\") pod \"f9bb5619-4bfe-49fb-a935-d867028f46b5\" (UID: \"f9bb5619-4bfe-49fb-a935-d867028f46b5\") "
Nov 01 20:56:17.864804 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864764    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:56:17.864804 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.864695    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:56:17.865268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.865246    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.865383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.865271    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.865383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.865285    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.865383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.865296    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb5619-4bfe-49fb-a935-d867028f46b5-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.865383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.865307    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.866779 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.866754    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:56:17.866958 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.866925    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:56:17.867029 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.866943    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "kube-api-access-cqwn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:56:17.868021 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.867995    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token" pod "f9bb5619-4bfe-49fb-a935-d867028f46b5" (UID: "f9bb5619-4bfe-49fb-a935-d867028f46b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:56:17.882326 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.882302    4872 generic.go:356] "Generic (PLEG): container finished" podID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerID="db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243" exitCode=0
Nov 01 20:56:17.882437 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.882363    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"
Nov 01 20:56:17.882437 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.882386    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" event={"ID":"f9bb5619-4bfe-49fb-a935-d867028f46b5","Type":"ContainerDied","Data":"db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243"}
Nov 01 20:56:17.882553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.882460    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg" event={"ID":"f9bb5619-4bfe-49fb-a935-d867028f46b5","Type":"ContainerDied","Data":"836f917e1fb4fae41260e796ccd104d8b43cb5fed39cc0bcfc1ab5254fee1466"}
Nov 01 20:56:17.882553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.882483    4872 scope.go:122] "RemoveContainer" containerID="db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243"
Nov 01 20:56:17.901306 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.901282    4872 scope.go:122] "RemoveContainer" containerID="db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243"
Nov 01 20:56:17.901616 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:56:17.901597    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243\": container with ID starting with db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243 not found: ID does not exist" containerID="db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243"
Nov 01 20:56:17.901708 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.901623    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243"} err="failed to get container status \"db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243\": rpc error: code = NotFound desc = could not find container \"db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243\": container with ID starting with db3552b337a95ebb668160ee451af13e312da4ff2546ceda59d5fd0f60e36243 not found: ID does not exist"
Nov 01 20:56:17.966339 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.966310    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.966339 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.966333    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.966339 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.966345    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:17.966540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:17.966354    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwn4\" (UniqueName: \"kubernetes.io/projected/f9bb5619-4bfe-49fb-a935-d867028f46b5-kube-api-access-cqwn4\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:56:18.095344 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:18.095260    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"]
Nov 01 20:56:18.099086 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:18.099057    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-hlgfg"]
Nov 01 20:56:19.083176 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:19.083140    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" path="/var/lib/kubelet/pods/f9bb5619-4bfe-49fb-a935-d867028f46b5/volumes"
Nov 01 20:56:23.365621 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:56:23.365577    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:56:26.935911 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.935869    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"]
Nov 01 20:56:26.936705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.936683    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy"
Nov 01 20:56:26.936864 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.936843    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy"
Nov 01 20:56:26.937076 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.937061    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bb5619-4bfe-49fb-a935-d867028f46b5" containerName="istio-proxy"
Nov 01 20:56:26.942552 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.942534    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963463 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963438    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlwn\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963568 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963484    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963633 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963598    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963771 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963758    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963812 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963801    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963914 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963903    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.963965 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.963927    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.964043 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.964029    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:26.964155 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:26.964141    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.038322 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.038291    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"]
Nov 01 20:56:27.045902 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.045879    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-24f923-openshift-default-dockercfg-sj525\""
Nov 01 20:56:27.065869 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.065842    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.065906    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlwn\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.065939    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.065982    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066162 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066110    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066168    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066267 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066247    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066322 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066277    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066322 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066288    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066413 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066367    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066813 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066787    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.066813 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.066796    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.067169 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.067147    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.067285 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.067222    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.068879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.068855    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.069070 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.069049    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.149684 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.146688    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.160798 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.155902    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlwn\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn\") pod \"e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") " pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.262178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.262081    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:27.702533 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.702498    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"]
Nov 01 20:56:27.703689 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:56:27.703639    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc56ebb_cc6f_467f_b6c5_46df5273f811.slice/crio-a77ceffeb117519c998434069befe45c59e67ca5c3eb51f2d93a10d0da0b44ff WatchSource:0}: Error finding container a77ceffeb117519c998434069befe45c59e67ca5c3eb51f2d93a10d0da0b44ff: Status 404 returned error can't find the container with id a77ceffeb117519c998434069befe45c59e67ca5c3eb51f2d93a10d0da0b44ff
Nov 01 20:56:27.706105 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.706073    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:56:27.706183 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.706136    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:56:27.706183 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:27.706163    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:56:28.019803 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:28.019699    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" event={"ID":"bcc56ebb-cc6f-467f-b6c5-46df5273f811","Type":"ContainerStarted","Data":"90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"}
Nov 01 20:56:28.019803 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:28.019763    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" event={"ID":"bcc56ebb-cc6f-467f-b6c5-46df5273f811","Type":"ContainerStarted","Data":"a77ceffeb117519c998434069befe45c59e67ca5c3eb51f2d93a10d0da0b44ff"}
Nov 01 20:56:28.141178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:28.141115    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podStartSLOduration=2.141102402 podStartE2EDuration="2.141102402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:28.137097634 +0000 UTC m=+1545.186207443" watchObservedRunningTime="2026-11-01 20:56:28.141102402 +0000 UTC m=+1545.190212218"
Nov 01 20:56:28.262875 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:28.262840    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:28.269406 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:28.269377    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:29.026922 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.026887    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:29.027863 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.027841    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:56:29.647055 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.647023    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"]
Nov 01 20:56:29.651986 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.651969    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.704421 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.704394    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a5ab4417-openshift-default-dockercfg-4tcv5\""
Nov 01 20:56:29.709416 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709386    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709497    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709585    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709619 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709608    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709813 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709623    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709813 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709800    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqt4\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709926 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709824    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709926 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709901    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.709926 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.709921    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.761724 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.761667    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"]
Nov 01 20:56:29.811515 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811485    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.811686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811523    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.811686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811541    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.811686 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811619    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqt4\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.811898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811649    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.811898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811819    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.811898 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811854    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812050 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811922    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812050 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.811977    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812050 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.812039    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812197 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.812148    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812284 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.812254    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812436 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.812410    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.812553 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.812413    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.814271 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.814248    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.814371 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.814330    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.869266 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.869213    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.869605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.869580    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqt4\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:29.967456 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:29.967395    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:30.400715 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:56:30.400685    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1432bbe5_a303_43a1_863f_3ff77f835e58.slice/crio-d768dce93962522ff5b0ce1d003b26d45849060e24dbe652dd1b7b8d9e2e3568 WatchSource:0}: Error finding container d768dce93962522ff5b0ce1d003b26d45849060e24dbe652dd1b7b8d9e2e3568: Status 404 returned error can't find the container with id d768dce93962522ff5b0ce1d003b26d45849060e24dbe652dd1b7b8d9e2e3568
Nov 01 20:56:30.401025 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:30.400764    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"]
Nov 01 20:56:30.403238 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:30.403205    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:56:30.403332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:30.403268    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:56:30.403332 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:30.403299    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:56:31.056406 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:31.056367    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" event={"ID":"1432bbe5-a303-43a1-863f-3ff77f835e58","Type":"ContainerStarted","Data":"82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01"}
Nov 01 20:56:31.056582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:31.056470    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" event={"ID":"1432bbe5-a303-43a1-863f-3ff77f835e58","Type":"ContainerStarted","Data":"d768dce93962522ff5b0ce1d003b26d45849060e24dbe652dd1b7b8d9e2e3568"}
Nov 01 20:56:31.309129 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:31.294155    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podStartSLOduration=2.294136394 podStartE2EDuration="2.294136394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:31.242875048 +0000 UTC m=+1548.291984865" watchObservedRunningTime="2026-11-01 20:56:31.294136394 +0000 UTC m=+1548.343246216"
Nov 01 20:56:31.968329 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:31.968300    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:31.969832 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:31.969805    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 20:56:31.969960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:31.969853    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 20:56:32.967879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:32.967844    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 20:56:32.968048 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:32.967910    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 20:56:33.967975 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:33.967941    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 20:56:33.968384 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:33.967996    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 20:56:34.967960 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:34.967928    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 20:56:34.968154 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:34.967995    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 20:56:35.967838 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:35.967802    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 20:56:35.968034 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:35.967855    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 20:56:36.972388 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:36.972361    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:36.972870 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:36.972766    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:36.973336 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:56:36.973314    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:56:53.367568 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:56:53.367531    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:57:14.996774 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:14.996702    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"]
Nov 01 20:57:15.001603 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.001578    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.112109 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.112065    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"]
Nov 01 20:57:15.176709 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.176644    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.176876 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.176758    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.176949 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.176928    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.176991 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.176976    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.177033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.177005    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.177111 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.177098    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.177252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.177236    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.177293 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.177265    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.177454 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.177431    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94ph\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279253    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279321 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279301    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279498 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279353    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279498 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279372    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279498 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279412    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94ph\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279602 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279565    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279651 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279607    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279733 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279702    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279733 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279607    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279820 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279757    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279820 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279761    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.279897 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.279819    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.280144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.280127    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.280212 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.280143    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.281948 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.281926    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.282113 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.282099    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.336972 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.336951    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.337062 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.337043    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94ph\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:15.616026 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:15.615948    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:16.052394 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:57:16.052352    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d300fd_d599_4aff_a404_b5ea6116a0cf.slice/crio-5db39c0859802154520e003f27cd056185699e50df3bc09eb801ee846cd272d7 WatchSource:0}: Error finding container 5db39c0859802154520e003f27cd056185699e50df3bc09eb801ee846cd272d7: Status 404 returned error can't find the container with id 5db39c0859802154520e003f27cd056185699e50df3bc09eb801ee846cd272d7
Nov 01 20:57:16.052788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.052538    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"]
Nov 01 20:57:16.055352 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.055320    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:57:16.055500 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.055401    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:57:16.055500 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.055446    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:57:16.639509 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.639476    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" event={"ID":"38d300fd-d599-4aff-a404-b5ea6116a0cf","Type":"ContainerStarted","Data":"78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4"}
Nov 01 20:57:16.639703 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.639536    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" event={"ID":"38d300fd-d599-4aff-a404-b5ea6116a0cf","Type":"ContainerStarted","Data":"5db39c0859802154520e003f27cd056185699e50df3bc09eb801ee846cd272d7"}
Nov 01 20:57:16.857091 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:16.857041    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podStartSLOduration=2.8570262 podStartE2EDuration="2.8570262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:16.806539203 +0000 UTC m=+1593.855649021" watchObservedRunningTime="2026-11-01 20:57:16.8570262 +0000 UTC m=+1593.906136015"
Nov 01 20:57:17.085164 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.085128    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"]
Nov 01 20:57:17.089537 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.089517    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.127623 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.127589    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrfr\" (UniqueName: \"kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.127820 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.127732    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.127889 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.127821    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.127931 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.127892    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.127967 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.127943    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.128040 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.128019    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.144799 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.144774    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-11a069b1\"/\"openshift-service-ca.crt\""
Nov 01 20:57:17.145204 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.145181    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-11a069b1\"/\"e2e-lifecycle-model-11a069b1-kserve-self-signed-certs\""
Nov 01 20:57:17.146228 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.146209    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-11a069b1\"/\"default-dockercfg-nlpgw\""
Nov 01 20:57:17.147627 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.147606    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-11a069b1\"/\"kube-root-ca.crt\""
Nov 01 20:57:17.162057 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.162035    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"]
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.278565    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrfr\" (UniqueName: \"kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.278699    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.278797    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.278871    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.278899    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.278941    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.279640    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289268 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.287413    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289807 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.289379    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.289807 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.289670    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.291675 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.291633    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.348925 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.348851    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrfr\" (UniqueName: \"kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr\") pod \"e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") " pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.433380 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.433348    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:17.640741 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.630737    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:17.645712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.644824    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:57:17.645712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.644886    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 20:57:17.854014 ip-10-0-131-188 kubenswrapper[4872]: W1101 20:57:17.853979    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod279f0ac6_d5c0_4962_85fe_d4d0d7c6ec28.slice/crio-c8745a0fb63cb9e4172800de111e2e14b22bc34e77af342570b02cf27ba668a3 WatchSource:0}: Error finding container c8745a0fb63cb9e4172800de111e2e14b22bc34e77af342570b02cf27ba668a3: Status 404 returned error can't find the container with id c8745a0fb63cb9e4172800de111e2e14b22bc34e77af342570b02cf27ba668a3
Nov 01 20:57:17.854197 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:17.854180    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"]
Nov 01 20:57:18.617678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:18.617110    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:57:18.617678 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:18.617181    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 20:57:18.672957 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:18.672919    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" event={"ID":"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28","Type":"ContainerStarted","Data":"9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82"}
Nov 01 20:57:18.673137 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:18.672970    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" event={"ID":"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28","Type":"ContainerStarted","Data":"c8745a0fb63cb9e4172800de111e2e14b22bc34e77af342570b02cf27ba668a3"}
Nov 01 20:57:18.790937 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:18.790885    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" podStartSLOduration=1.790869846 podStartE2EDuration="1.790869846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:18.788399181 +0000 UTC m=+1595.837508998" watchObservedRunningTime="2026-11-01 20:57:18.790869846 +0000 UTC m=+1595.839979678"
Nov 01 20:57:19.433965 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:19.433929    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:19.619110 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:19.619075    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:57:19.619527 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:19.619140    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 20:57:20.618681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:20.618086    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:57:20.618681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:20.618184    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 20:57:21.620830 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:21.620786    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:57:21.621211 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:21.620849    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 20:57:21.751867 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:21.751831    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"]
Nov 01 20:57:21.868328 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:21.868292    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"]
Nov 01 20:57:22.621319 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:22.621286    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:22.621728 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:22.621565    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:22.622314 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:22.622292    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:57:23.414334 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:57:23.414295    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:57:26.753186 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:26.753132    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76" podUID="da74b6a9-6e87-4818-8cc9-b64e9da8403b" containerName="payload-processing" containerID="cri-o://5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d" gracePeriod=25
Nov 01 20:57:26.868819 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:26.868765    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2" podUID="83e8391f-cf8b-4f90-8e0b-c130634ee2c3" containerName="payload-pre-processing" containerID="cri-o://043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31" gracePeriod=25
Nov 01 20:57:27.140868 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.139948    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:57:27.256811 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.256782    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb\" (UniqueName: \"kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb\") pod \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") "
Nov 01 20:57:27.257054 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.257037    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume\") pod \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\" (UID: \"da74b6a9-6e87-4818-8cc9-b64e9da8403b\") "
Nov 01 20:57:27.257511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.257483    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume" pod "da74b6a9-6e87-4818-8cc9-b64e9da8403b" (UID: "da74b6a9-6e87-4818-8cc9-b64e9da8403b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:27.258301 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.258275    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da74b6a9-6e87-4818-8cc9-b64e9da8403b-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:27.259383 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.259361    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb" pod "da74b6a9-6e87-4818-8cc9-b64e9da8403b" (UID: "da74b6a9-6e87-4818-8cc9-b64e9da8403b"). InnerVolumeSpecName "kube-api-access-58spb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:27.261160 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.261135    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:57:27.359869 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.359768    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume\") pod \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") "
Nov 01 20:57:27.360027 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.359980    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp\" (UniqueName: \"kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp\") pod \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\" (UID: \"83e8391f-cf8b-4f90-8e0b-c130634ee2c3\") "
Nov 01 20:57:27.360157 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.360132    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume" pod "83e8391f-cf8b-4f90-8e0b-c130634ee2c3" (UID: "83e8391f-cf8b-4f90-8e0b-c130634ee2c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:27.360879 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.360855    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58spb\" (UniqueName: \"kubernetes.io/projected/da74b6a9-6e87-4818-8cc9-b64e9da8403b-kube-api-access-58spb\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:27.360993 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.360881    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:27.362201 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.362175    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp" pod "83e8391f-cf8b-4f90-8e0b-c130634ee2c3" (UID: "83e8391f-cf8b-4f90-8e0b-c130634ee2c3"). InnerVolumeSpecName "kube-api-access-7zrwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:27.436103 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.436062    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:27.469282 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.469249    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zrwp\" (UniqueName: \"kubernetes.io/projected/83e8391f-cf8b-4f90-8e0b-c130634ee2c3-kube-api-access-7zrwp\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:27.469472 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.469352    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:27.744966 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.744932    4872 generic.go:356] "Generic (PLEG): container finished" podID="83e8391f-cf8b-4f90-8e0b-c130634ee2c3" containerID="043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31" exitCode=0
Nov 01 20:57:27.745165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.745000    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"
Nov 01 20:57:27.745165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.745016    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2" event={"ID":"83e8391f-cf8b-4f90-8e0b-c130634ee2c3","Type":"ContainerDied","Data":"043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31"}
Nov 01 20:57:27.745165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.745073    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2" event={"ID":"83e8391f-cf8b-4f90-8e0b-c130634ee2c3","Type":"ContainerDied","Data":"31a9997c23406dc30d7fbb33a54c0dfac356b103ec060242fa1668a9e0f62ee0"}
Nov 01 20:57:27.745165 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.745102    4872 scope.go:122] "RemoveContainer" containerID="043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31"
Nov 01 20:57:27.746448 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.746422    4872 generic.go:356] "Generic (PLEG): container finished" podID="da74b6a9-6e87-4818-8cc9-b64e9da8403b" containerID="5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d" exitCode=0
Nov 01 20:57:27.746512 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.746479    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"
Nov 01 20:57:27.746512 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.746481    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76" event={"ID":"da74b6a9-6e87-4818-8cc9-b64e9da8403b","Type":"ContainerDied","Data":"5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d"}
Nov 01 20:57:27.746585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.746529    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76" event={"ID":"da74b6a9-6e87-4818-8cc9-b64e9da8403b","Type":"ContainerDied","Data":"0b501dc5a3d98ac814101855941b9dd98f3e7bd40d76cc356b5d12982c0afa14"}
Nov 01 20:57:27.757582 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.757255    4872 scope.go:122] "RemoveContainer" containerID="043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31"
Nov 01 20:57:27.758467 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:57:27.758441    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31\": container with ID starting with 043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31 not found: ID does not exist" containerID="043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31"
Nov 01 20:57:27.758467 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.758460    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 20:57:27.758584 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.758472    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31"} err="failed to get container status \"043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31\": rpc error: code = NotFound desc = could not find container \"043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31\": container with ID starting with 043f2255e45418367da054c426c11ef094cadc1e880fcde36e60ddcc8fb6af31 not found: ID does not exist"
Nov 01 20:57:27.758584 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.758489    4872 scope.go:122] "RemoveContainer" containerID="5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d"
Nov 01 20:57:27.781851 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.781831    4872 scope.go:122] "RemoveContainer" containerID="5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d"
Nov 01 20:57:27.782182 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:57:27.782161    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d\": container with ID starting with 5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d not found: ID does not exist" containerID="5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d"
Nov 01 20:57:27.782240 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.782193    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d"} err="failed to get container status \"5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d\": rpc error: code = NotFound desc = could not find container \"5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d\": container with ID starting with 5cc53ea8e1a45a2f26e3b10380ee5d9f5484523c53df5810e250ac8e8b50ac6d not found: ID does not exist"
Nov 01 20:57:27.932136 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.932103    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"]
Nov 01 20:57:27.937881 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:27.937852    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5ab4417-5677fb94bc-mg6v2"]
Nov 01 20:57:28.167307 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.167275    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"]
Nov 01 20:57:28.174685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.174646    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5ab4417-6fcc7f8d97-gpp76"]
Nov 01 20:57:28.377350 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.377316    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"]
Nov 01 20:57:28.377581 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.377555    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" podUID="42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" containerName="maas-api" containerID="cri-o://68ff60448bdfd30f4877b6f9933a005c5a32152bb5ad7a9bc37d0cc0a506b898" gracePeriod=30
Nov 01 20:57:28.764605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.764514    4872 generic.go:356] "Generic (PLEG): container finished" podID="42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" containerID="68ff60448bdfd30f4877b6f9933a005c5a32152bb5ad7a9bc37d0cc0a506b898" exitCode=0
Nov 01 20:57:28.765923 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.765850    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" event={"ID":"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1","Type":"ContainerDied","Data":"68ff60448bdfd30f4877b6f9933a005c5a32152bb5ad7a9bc37d0cc0a506b898"}
Nov 01 20:57:28.963702 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:28.962517    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:57:29.071780 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.071706    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd\" (UniqueName: \"kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd\") pod \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") "
Nov 01 20:57:29.071916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.071806    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\") pod \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") "
Nov 01 20:57:29.071916 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.071848    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs\") pod \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") "
Nov 01 20:57:29.072042 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.072023    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca\") pod \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\" (UID: \"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1\") "
Nov 01 20:57:29.072415 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.072389    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca" pod "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" (UID: "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:29.072739 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.072716    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:29.074178 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.074150    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd" pod "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" (UID: "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1"). InnerVolumeSpecName "kube-api-access-z6kfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:29.074444 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.074413    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs" pod "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" (UID: "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:29.074444 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.074424    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls" pod "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" (UID: "42e36f34-a9f1-4ac4-929a-c4c8c09d55b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:29.087213 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.087172    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e8391f-cf8b-4f90-8e0b-c130634ee2c3" path="/var/lib/kubelet/pods/83e8391f-cf8b-4f90-8e0b-c130634ee2c3/volumes"
Nov 01 20:57:29.087920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.087895    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da74b6a9-6e87-4818-8cc9-b64e9da8403b" path="/var/lib/kubelet/pods/da74b6a9-6e87-4818-8cc9-b64e9da8403b/volumes"
Nov 01 20:57:29.175257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.174993    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6kfd\" (UniqueName: \"kubernetes.io/projected/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-kube-api-access-z6kfd\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:29.175257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.175019    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:29.175257 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.175028    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:29.573993 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.573956    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"]
Nov 01 20:57:29.574243 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.574218    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" podUID="de25f9c8-2083-497b-9547-d6688fb5e7e7" containerName="maas-api" containerID="cri-o://ece9eafdf5a98f91c4377618e0fa746552631aaf3ec5b24ae90ebde61e787274" gracePeriod=30
Nov 01 20:57:29.775033 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.775004    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"
Nov 01 20:57:29.775520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.775028    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c" event={"ID":"42e36f34-a9f1-4ac4-929a-c4c8c09d55b1","Type":"ContainerDied","Data":"ba3b5b90026053c499958612cbd449862d5edeac9500942dbfef410f4fc22ddb"}
Nov 01 20:57:29.775520 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.775099    4872 scope.go:122] "RemoveContainer" containerID="68ff60448bdfd30f4877b6f9933a005c5a32152bb5ad7a9bc37d0cc0a506b898"
Nov 01 20:57:29.841791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.814561    4872 generic.go:356] "Generic (PLEG): container finished" podID="de25f9c8-2083-497b-9547-d6688fb5e7e7" containerID="ece9eafdf5a98f91c4377618e0fa746552631aaf3ec5b24ae90ebde61e787274" exitCode=0
Nov 01 20:57:29.852823 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:29.846781    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" event={"ID":"de25f9c8-2083-497b-9547-d6688fb5e7e7","Type":"ContainerDied","Data":"ece9eafdf5a98f91c4377618e0fa746552631aaf3ec5b24ae90ebde61e787274"}
Nov 01 20:57:30.197559 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.197494    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"]
Nov 01 20:57:30.208681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.207634    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ddfb06d-5584bb94d-c468c"]
Nov 01 20:57:30.241581 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.241543    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:57:30.382414 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.382316    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg\" (UniqueName: \"kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg\") pod \"de25f9c8-2083-497b-9547-d6688fb5e7e7\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") "
Nov 01 20:57:30.382585 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.382414    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\") pod \"de25f9c8-2083-497b-9547-d6688fb5e7e7\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") "
Nov 01 20:57:30.382674 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.382581    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs\") pod \"de25f9c8-2083-497b-9547-d6688fb5e7e7\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") "
Nov 01 20:57:30.382745 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.382683    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca\") pod \"de25f9c8-2083-497b-9547-d6688fb5e7e7\" (UID: \"de25f9c8-2083-497b-9547-d6688fb5e7e7\") "
Nov 01 20:57:30.383280 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.383247    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca" pod "de25f9c8-2083-497b-9547-d6688fb5e7e7" (UID: "de25f9c8-2083-497b-9547-d6688fb5e7e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:30.383562 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.383534    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de25f9c8-2083-497b-9547-d6688fb5e7e7-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:30.386808 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.386780    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs" pod "de25f9c8-2083-497b-9547-d6688fb5e7e7" (UID: "de25f9c8-2083-497b-9547-d6688fb5e7e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:30.391865 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.391831    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg" pod "de25f9c8-2083-497b-9547-d6688fb5e7e7" (UID: "de25f9c8-2083-497b-9547-d6688fb5e7e7"). InnerVolumeSpecName "kube-api-access-slpxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:30.393876 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.393842    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls" pod "de25f9c8-2083-497b-9547-d6688fb5e7e7" (UID: "de25f9c8-2083-497b-9547-d6688fb5e7e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:30.484712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.484675    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:30.484712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.484708    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slpxg\" (UniqueName: \"kubernetes.io/projected/de25f9c8-2083-497b-9547-d6688fb5e7e7-kube-api-access-slpxg\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:30.484712 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.484719    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de25f9c8-2083-497b-9547-d6688fb5e7e7-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:30.859394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.859365    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"
Nov 01 20:57:30.859394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.859377    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l" event={"ID":"de25f9c8-2083-497b-9547-d6688fb5e7e7","Type":"ContainerDied","Data":"8e81aa3b56aa326fcde4a5508d6e55b568d015836d8b914e9f720a96aaba6343"}
Nov 01 20:57:30.859959 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:30.859477    4872 scope.go:122] "RemoveContainer" containerID="ece9eafdf5a98f91c4377618e0fa746552631aaf3ec5b24ae90ebde61e787274"
Nov 01 20:57:31.037614 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.037577    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"]
Nov 01 20:57:31.088326 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.088287    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" path="/var/lib/kubelet/pods/42e36f34-a9f1-4ac4-929a-c4c8c09d55b1/volumes"
Nov 01 20:57:31.088797 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.088778    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74ab02fe-777d6bd454-j899l"]
Nov 01 20:57:31.379102 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.379058    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"]
Nov 01 20:57:31.380287 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380256    4872 cpu_manager.go:414] "containerMap: removing container" podUID="de25f9c8-2083-497b-9547-d6688fb5e7e7" containerName="maas-api"
Nov 01 20:57:31.380557 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380539    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de25f9c8-2083-497b-9547-d6688fb5e7e7" containerName="maas-api"
Nov 01 20:57:31.380710 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380691    4872 cpu_manager.go:414] "containerMap: removing container" podUID="42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" containerName="maas-api"
Nov 01 20:57:31.380710 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380707    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" containerName="maas-api"
Nov 01 20:57:31.380862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380746    4872 cpu_manager.go:414] "containerMap: removing container" podUID="da74b6a9-6e87-4818-8cc9-b64e9da8403b" containerName="payload-processing"
Nov 01 20:57:31.380862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380754    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da74b6a9-6e87-4818-8cc9-b64e9da8403b" containerName="payload-processing"
Nov 01 20:57:31.380862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380802    4872 cpu_manager.go:414] "containerMap: removing container" podUID="83e8391f-cf8b-4f90-8e0b-c130634ee2c3" containerName="payload-pre-processing"
Nov 01 20:57:31.380862 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.380810    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e8391f-cf8b-4f90-8e0b-c130634ee2c3" containerName="payload-pre-processing"
Nov 01 20:57:31.381021 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.381007    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e8391f-cf8b-4f90-8e0b-c130634ee2c3" containerName="payload-pre-processing"
Nov 01 20:57:31.381161 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.381148    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="de25f9c8-2083-497b-9547-d6688fb5e7e7" containerName="maas-api"
Nov 01 20:57:31.381247 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.381238    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e36f34-a9f1-4ac4-929a-c4c8c09d55b1" containerName="maas-api"
Nov 01 20:57:31.381341 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.381331    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="da74b6a9-6e87-4818-8cc9-b64e9da8403b" containerName="payload-processing"
Nov 01 20:57:31.388137 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.388115    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.437222 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.437190    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"]
Nov 01 20:57:31.440831 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.440802    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-24f923-serving-cert\""
Nov 01 20:57:31.514295 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.514260    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.514486 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.514324    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwxs\" (UniqueName: \"kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.514486 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.514390    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.514985 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.514755    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.616281 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.616241    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwxs\" (UniqueName: \"kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.616465 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.616324    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.616622 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.616604    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.616748 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.616733    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.616909 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:57:31.616886    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-serving-cert: secret "maas-api-e2e-shared-a-w3-24f923-serving-cert" not found
Nov 01 20:57:31.616993 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:57:31.616980    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls podName:66f7599d-77fe-41d6-a9e6-2be4a2515e02 nodeName:}" failed. No retries permitted until 2026-11-01 20:57:32.116959244 +0000 UTC m=+1609.166069047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls") pod "maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" (UID: "66f7599d-77fe-41d6-a9e6-2be4a2515e02") : secret "maas-api-e2e-shared-a-w3-24f923-serving-cert" not found
Nov 01 20:57:31.617136 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.617117    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.619690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.619634    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:31.674776 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:31.674747    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwxs\" (UniqueName: \"kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:32.205185 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:32.205133    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:32.232511 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:32.232477    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:32.307787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:32.307756    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:32.886989 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:32.876175    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"]
Nov 01 20:57:32.951294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:32.951227    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" event={"ID":"66f7599d-77fe-41d6-a9e6-2be4a2515e02","Type":"ContainerStarted","Data":"93a4cbfd0ac4de852924d25f6b7110f4d8b765c73ed01589abf40f1b95b48699"}
Nov 01 20:57:33.091560 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:33.091522    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de25f9c8-2083-497b-9547-d6688fb5e7e7" path="/var/lib/kubelet/pods/de25f9c8-2083-497b-9547-d6688fb5e7e7/volumes"
Nov 01 20:57:34.976522 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:34.976469    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" event={"ID":"66f7599d-77fe-41d6-a9e6-2be4a2515e02","Type":"ContainerStarted","Data":"a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537"}
Nov 01 20:57:34.977149 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:34.977111    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:35.099343 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.099263    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" podStartSLOduration=3.1958368950000002 podStartE2EDuration="4.099241362s" totalImagesPullingTime="903.404467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:35.096813438 +0000 UTC m=+1612.145923266" watchObservedRunningTime="2026-11-01 20:57:35.099241362 +0000 UTC m=+1612.148351184"
Nov 01 20:57:35.986377 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.986347    4872 generic.go:356] "Generic (PLEG): container finished" podID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerID="1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde" exitCode=0
Nov 01 20:57:35.986794 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.986422    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" event={"ID":"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f","Type":"ContainerDied","Data":"1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"}
Nov 01 20:57:35.986928 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.986910    4872 scope.go:122] "RemoveContainer" containerID="1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"
Nov 01 20:57:35.988316 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.988288    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:57:35.988419 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.988353    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:57:35.988419 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:35.988388    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:57:37.023078 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:37.023042    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" event={"ID":"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f","Type":"ContainerStarted","Data":"0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f"}
Nov 01 20:57:37.445440 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:37.445331    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:57:37.447696 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:37.446601    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:57:37.447696 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:37.446684    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:57:38.446011 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:38.445974    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:57:38.446425 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:38.446039    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:57:39.446456 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:39.446424    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:57:39.446877 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:39.446487    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:57:39.649464 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:39.649424    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:57:40.446294 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:40.446260    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:57:40.446469 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:40.446310    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:57:41.020897 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:41.020862    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 20:57:41.465681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:41.457988    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:57:41.465681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:41.458048    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:57:42.470695 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:42.470621    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:57:42.472195 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:42.472169    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 20:57:53.366371 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:57:53.366330    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:57:58.009974 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.009937    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"]
Nov 01 20:57:58.010812 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.010759    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" podUID="db52d2a0-5f81-4a2b-8172-4de59b3e7eab" containerName="maas-api" containerID="cri-o://ab0ccdf77136be8a78431e1fa0b97858042fe4f61bb1494ff0c69cf0d6811d8a" gracePeriod=30
Nov 01 20:57:58.315761 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.311864    4872 generic.go:356] "Generic (PLEG): container finished" podID="db52d2a0-5f81-4a2b-8172-4de59b3e7eab" containerID="ab0ccdf77136be8a78431e1fa0b97858042fe4f61bb1494ff0c69cf0d6811d8a" exitCode=0
Nov 01 20:57:58.315761 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.312058    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" event={"ID":"db52d2a0-5f81-4a2b-8172-4de59b3e7eab","Type":"ContainerDied","Data":"ab0ccdf77136be8a78431e1fa0b97858042fe4f61bb1494ff0c69cf0d6811d8a"}
Nov 01 20:57:58.373091 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.373058    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:57:58.509915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.507235    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs\") pod \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") "
Nov 01 20:57:58.509915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.507317    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\") pod \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") "
Nov 01 20:57:58.509915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.507589    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca\") pod \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") "
Nov 01 20:57:58.509915 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.507804    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc\" (UniqueName: \"kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc\") pod \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\" (UID: \"db52d2a0-5f81-4a2b-8172-4de59b3e7eab\") "
Nov 01 20:57:58.518060 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.510681    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca" pod "db52d2a0-5f81-4a2b-8172-4de59b3e7eab" (UID: "db52d2a0-5f81-4a2b-8172-4de59b3e7eab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:58.536951 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.535366    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc" pod "db52d2a0-5f81-4a2b-8172-4de59b3e7eab" (UID: "db52d2a0-5f81-4a2b-8172-4de59b3e7eab"). InnerVolumeSpecName "kube-api-access-m7wdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:58.536951 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.535857    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls" pod "db52d2a0-5f81-4a2b-8172-4de59b3e7eab" (UID: "db52d2a0-5f81-4a2b-8172-4de59b3e7eab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:58.536951 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.536306    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs" pod "db52d2a0-5f81-4a2b-8172-4de59b3e7eab" (UID: "db52d2a0-5f81-4a2b-8172-4de59b3e7eab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:58.611063 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.610961    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:58.611063 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.611001    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:58.611063 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.611017    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:58.611063 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:58.611029    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wdc\" (UniqueName: \"kubernetes.io/projected/db52d2a0-5f81-4a2b-8172-4de59b3e7eab-kube-api-access-m7wdc\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:57:59.321049 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.321020    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"
Nov 01 20:57:59.321466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.321044    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l" event={"ID":"db52d2a0-5f81-4a2b-8172-4de59b3e7eab","Type":"ContainerDied","Data":"229820a368703c4921c2905d9beaf7b41436e7c7a4172aa37ebd7788ae8dbe1d"}
Nov 01 20:57:59.321466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.321107    4872 scope.go:122] "RemoveContainer" containerID="ab0ccdf77136be8a78431e1fa0b97858042fe4f61bb1494ff0c69cf0d6811d8a"
Nov 01 20:57:59.352600 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.352575    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"]
Nov 01 20:57:59.352806 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.352786    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" containerID="cri-o://73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8" gracePeriod=30
Nov 01 20:57:59.513588 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.513499    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"]
Nov 01 20:57:59.517349 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:57:59.517320    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a5ab4417-6494d5595f-w276l"]
Nov 01 20:58:01.080943 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:01.080913    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db52d2a0-5f81-4a2b-8172-4de59b3e7eab" path="/var/lib/kubelet/pods/db52d2a0-5f81-4a2b-8172-4de59b3e7eab/volumes"
Nov 01 20:58:04.034751 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:04.034710    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:58:04.035387 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:04.035352    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:58:05.080394 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.080371    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:58:05.219229 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219189    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219425 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219251    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219425 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219283    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219425 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219392    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219605 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219548    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219705 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219626    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219886 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219865    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.219954 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219910    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.220012 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.219980    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket\") pod \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\" (UID: \"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0\") "
Nov 01 20:58:05.220495 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.220463    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:05.220717 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.220691    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:05.221765 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.221394    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:05.221909 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.221887    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:05.222038 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.222015    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:05.222446 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.222395    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.222446 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.222419    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.222446 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.222433    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.222446 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.222447    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.222736 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.222460    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.223283 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.223255    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:05.224360 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.224333    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:58:05.224360 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.224340    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:05.224502 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.224392    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq" pod "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" (UID: "b70c2a48-02a0-47ab-8d9c-52b3d3f590a0"). InnerVolumeSpecName "kube-api-access-7zmxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:05.324256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.324214    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.324256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.324255    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.324496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.324271    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmxq\" (UniqueName: \"kubernetes.io/projected/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-kube-api-access-7zmxq\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.324496 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.324287    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:05.392599 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.392565    4872 generic.go:356] "Generic (PLEG): container finished" podID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerID="73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8" exitCode=0
Nov 01 20:58:05.393116 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.392765    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" event={"ID":"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0","Type":"ContainerDied","Data":"73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8"}
Nov 01 20:58:05.393310 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.393285    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x" event={"ID":"b70c2a48-02a0-47ab-8d9c-52b3d3f590a0","Type":"ContainerDied","Data":"2b2374418083ffc00683f30713475f327d1cc74ed1ad6550501f5aab2d3aba83"}
Nov 01 20:58:05.393440 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.393425    4872 scope.go:122] "RemoveContainer" containerID="73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8"
Nov 01 20:58:05.421788 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.421713    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"
Nov 01 20:58:05.443920 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.443898    4872 scope.go:122] "RemoveContainer" containerID="73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8"
Nov 01 20:58:05.444275 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:58:05.444249    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8\": container with ID starting with 73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8 not found: ID does not exist" containerID="73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8"
Nov 01 20:58:05.444369 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.444281    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8"} err="failed to get container status \"73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8\": rpc error: code = NotFound desc = could not find container \"73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8\": container with ID starting with 73c6ab9f3c220d97d0f0a5646c4175c230282f867e8477833ece73ce27b784b8 not found: ID does not exist"
Nov 01 20:58:05.593221 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.593130    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"]
Nov 01 20:58:05.627252 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:05.627211    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-wmg5x"]
Nov 01 20:58:07.032974 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.032944    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"]
Nov 01 20:58:07.033474 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.033162    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" containerID="cri-o://78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4" gracePeriod=30
Nov 01 20:58:07.039022 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.038994    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"]
Nov 01 20:58:07.039231 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.039205    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy" containerID="cri-o://82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01" gracePeriod=30
Nov 01 20:58:07.087409 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.087341    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" path="/var/lib/kubelet/pods/b70c2a48-02a0-47ab-8d9c-52b3d3f590a0/volumes"
Nov 01 20:58:07.624389 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.624346    4872 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:58:07.624541 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:07.624414    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:58:12.352887 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.352855    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:58:12.372723 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.372702    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:58:12.426540 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426515    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.426681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426573    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.426681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426643    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.426795 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426701    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.426795 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426732    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.426795 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426761    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.426941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426820    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.426941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426856    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.426941 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426898    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427107 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.426969    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:12.427107 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427077    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427107 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427056    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.427256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427161    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427207    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.427256 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427239    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.427403 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427276    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.427403 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427292    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427403 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427366    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427556 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427411    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427556 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427474    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data\") pod \"1432bbe5-a303-43a1-863f-3ff77f835e58\" (UID: \"1432bbe5-a303-43a1-863f-3ff77f835e58\") "
Nov 01 20:58:12.427556 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427551    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo\") pod \"38d300fd-d599-4aff-a404-b5ea6116a0cf\" (UID: \"38d300fd-d599-4aff-a404-b5ea6116a0cf\") "
Nov 01 20:58:12.427725 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427617    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.427855 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427815    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.427921 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.427891    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.428023 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428000    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.428397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428181    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.428397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428333    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.428397 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428360    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428520    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d300fd-d599-4aff-a404-b5ea6116a0cf-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428538    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428551    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428565    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428581    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428596    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.428607 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428609    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432bbe5-a303-43a1-863f-3ff77f835e58-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.429338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428623    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.429338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428637    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.429338 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.428649    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.429690 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.429651    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:12.430248 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.430222    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:58:12.430492 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.430463    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.430579 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.430490    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:12.430832 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.430810    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "kube-api-access-4kqt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:12.430933 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.430920    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "kube-api-access-z94ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:12.431009 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.430986    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy" pod "38d300fd-d599-4aff-a404-b5ea6116a0cf" (UID: "38d300fd-d599-4aff-a404-b5ea6116a0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:12.431313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.431297    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo" pod "1432bbe5-a303-43a1-863f-3ff77f835e58" (UID: "1432bbe5-a303-43a1-863f-3ff77f835e58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:58:12.481077 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.481049    4872 generic.go:356] "Generic (PLEG): container finished" podID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerID="78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4" exitCode=0
Nov 01 20:58:12.481174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.481100    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"
Nov 01 20:58:12.481174 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.481143    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" event={"ID":"38d300fd-d599-4aff-a404-b5ea6116a0cf","Type":"ContainerDied","Data":"78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4"}
Nov 01 20:58:12.481261 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.481217    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w" event={"ID":"38d300fd-d599-4aff-a404-b5ea6116a0cf","Type":"ContainerDied","Data":"5db39c0859802154520e003f27cd056185699e50df3bc09eb801ee846cd272d7"}
Nov 01 20:58:12.481261 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.481241    4872 scope.go:122] "RemoveContainer" containerID="78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4"
Nov 01 20:58:12.483466 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.483442    4872 generic.go:356] "Generic (PLEG): container finished" podID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerID="82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01" exitCode=0
Nov 01 20:58:12.483780 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.483499    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"
Nov 01 20:58:12.483780 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.483537    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" event={"ID":"1432bbe5-a303-43a1-863f-3ff77f835e58","Type":"ContainerDied","Data":"82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01"}
Nov 01 20:58:12.483780 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.483618    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw" event={"ID":"1432bbe5-a303-43a1-863f-3ff77f835e58","Type":"ContainerDied","Data":"d768dce93962522ff5b0ce1d003b26d45849060e24dbe652dd1b7b8d9e2e3568"}
Nov 01 20:58:12.496648 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.496612    4872 scope.go:122] "RemoveContainer" containerID="78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4"
Nov 01 20:58:12.497109 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:58:12.496964    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4\": container with ID starting with 78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4 not found: ID does not exist" containerID="78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4"
Nov 01 20:58:12.497109 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.497003    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4"} err="failed to get container status \"78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4\": rpc error: code = NotFound desc = could not find container \"78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4\": container with ID starting with 78ca2e124c911dc97cecf612a0c4199dfd02173d6bad1be321dd75b09306dfa4 not found: ID does not exist"
Nov 01 20:58:12.497109 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.497027    4872 scope.go:122] "RemoveContainer" containerID="82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01"
Nov 01 20:58:12.511467 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.511452    4872 scope.go:122] "RemoveContainer" containerID="82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01"
Nov 01 20:58:12.511817 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:58:12.511789    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01\": container with ID starting with 82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01 not found: ID does not exist" containerID="82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01"
Nov 01 20:58:12.511956 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.511826    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01"} err="failed to get container status \"82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01\": rpc error: code = NotFound desc = could not find container \"82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01\": container with ID starting with 82639d3c1cab14c5b28a008a99e015727808f86371816a5f76eace51e23c2c01 not found: ID does not exist"
Nov 01 20:58:12.530032 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530006    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530032 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530029    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94ph\" (UniqueName: \"kubernetes.io/projected/38d300fd-d599-4aff-a404-b5ea6116a0cf-kube-api-access-z94ph\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530194 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530041    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530194 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530050    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530194 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530063    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530194 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530074    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kqt4\" (UniqueName: \"kubernetes.io/projected/1432bbe5-a303-43a1-863f-3ff77f835e58-kube-api-access-4kqt4\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530194 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530083    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d300fd-d599-4aff-a404-b5ea6116a0cf-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.530194 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.530090    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432bbe5-a303-43a1-863f-3ff77f835e58-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:58:12.727583 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.727536    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"]
Nov 01 20:58:12.733262 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.733235    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-kztdw"]
Nov 01 20:58:12.838791 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.838758    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"]
Nov 01 20:58:12.842091 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:12.842071    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-9tm7w"]
Nov 01 20:58:13.081787 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.081751    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" path="/var/lib/kubelet/pods/1432bbe5-a303-43a1-863f-3ff77f835e58/volumes"
Nov 01 20:58:13.082566 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.082546    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" path="/var/lib/kubelet/pods/38d300fd-d599-4aff-a404-b5ea6116a0cf/volumes"
Nov 01 20:58:13.493145 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.493102    4872 generic.go:356] "Generic (PLEG): container finished" podID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerID="90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a" exitCode=0
Nov 01 20:58:13.493538 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.493215    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" event={"ID":"bcc56ebb-cc6f-467f-b6c5-46df5273f811","Type":"ContainerDied","Data":"90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"}
Nov 01 20:58:13.493639 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.493618    4872 scope.go:122] "RemoveContainer" containerID="90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"
Nov 01 20:58:13.495400 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.495373    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:58:13.495471 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.495436    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:58:13.495506 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:13.495477    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:58:14.031679 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:14.028227    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:58:14.527313 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:14.527276    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" event={"ID":"bcc56ebb-cc6f-467f-b6c5-46df5273f811","Type":"ContainerStarted","Data":"c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455"}
Nov 01 20:58:15.263271 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:15.263244    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:58:15.264487 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:15.264463    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:58:15.264609 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:15.264519    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:58:16.262763 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:16.262732    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:58:16.263144 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:16.262787    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:58:17.267524 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:17.266526    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:58:17.267524 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:17.266626    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:58:18.262474 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:18.262438    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:58:18.262685 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:18.262499    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:58:19.262849 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:19.262818    4872 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:58:19.263230 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:19.262875    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:58:20.266681 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:20.266638    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:58:20.267172 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:20.267156    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:58:20.267831 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:58:20.267806    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 20:58:23.366226 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:58:23.366191    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:58:53.376936 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:58:53.376890    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:59:23.366131 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:59:23.366097    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:59:53.366299 ip-10-0-131-188 kubenswrapper[4872]: E1101 20:59:53.366264    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 20:59:58.478142 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:58.478066    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"]
Nov 01 20:59:58.478562 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:58.478286    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" containerID="cri-o://8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466" gracePeriod=30
Nov 01 20:59:59.177641 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.177609    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:59:59.177847 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.177700    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:59:59.238989 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.238957    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"]
Nov 01 20:59:59.239597 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.239170    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy" containerID="cri-o://f64ce4e70706638491d92cf3ab7f1e9e51128bbac27040423a873c4f42362cb1" gracePeriod=30
Nov 01 20:59:59.538146 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.538070    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"]
Nov 01 20:59:59.538559 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.538258    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy" containerID="cri-o://ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934" gracePeriod=30
Nov 01 20:59:59.909411 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.909382    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"]
Nov 01 20:59:59.909596 ip-10-0-131-188 kubenswrapper[4872]: I1101 20:59:59.909573    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" containerID="cri-o://8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb" gracePeriod=30
Nov 01 21:00:00.192036 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.191966    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"]
Nov 01 21:00:00.192634 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192621    4872 cpu_manager.go:414] "containerMap: removing container" podUID="db52d2a0-5f81-4a2b-8172-4de59b3e7eab" containerName="maas-api"
Nov 01 21:00:00.192709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192635    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db52d2a0-5f81-4a2b-8172-4de59b3e7eab" containerName="maas-api"
Nov 01 21:00:00.192709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192650    4872 cpu_manager.go:414] "containerMap: removing container" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy"
Nov 01 21:00:00.192709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192672    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy"
Nov 01 21:00:00.192709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192687    4872 cpu_manager.go:414] "containerMap: removing container" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy"
Nov 01 21:00:00.192709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192692    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy"
Nov 01 21:00:00.192709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192709    4872 cpu_manager.go:414] "containerMap: removing container" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy"
Nov 01 21:00:00.192884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192714    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy"
Nov 01 21:00:00.192884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192835    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70c2a48-02a0-47ab-8d9c-52b3d3f590a0" containerName="istio-proxy"
Nov 01 21:00:00.192884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192845    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="1432bbe5-a303-43a1-863f-3ff77f835e58" containerName="istio-proxy"
Nov 01 21:00:00.192884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192856    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="db52d2a0-5f81-4a2b-8172-4de59b3e7eab" containerName="maas-api"
Nov 01 21:00:00.192884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.192863    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d300fd-d599-4aff-a404-b5ea6116a0cf" containerName="istio-proxy"
Nov 01 21:00:00.196739 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.196716    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.215329 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.215301    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.215491 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.215471    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkfx\" (UniqueName: \"kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx\") pod \"maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.216526 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.216504    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"]
Nov 01 21:00:00.221219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.221197    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"]
Nov 01 21:00:00.221816 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.221794    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.226881 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.226863    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.238843 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.238819    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"]
Nov 01 21:00:00.244292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.244267    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.264742 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.264717    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"]
Nov 01 21:00:00.277317 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.277285    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"]
Nov 01 21:00:00.293065 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.293042    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"]
Nov 01 21:00:00.316614 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.316583    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhcg\" (UniqueName: \"kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.316770 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.316666    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.316770 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.316739    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.316878 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.316774    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qdh\" (UniqueName: \"kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.316931 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.316904    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkfx\" (UniqueName: \"kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx\") pod \"maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.316983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.316941    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2hd\" (UniqueName: \"kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.317038 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.317007    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.317114 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.317100    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.317844 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.317821    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.347337 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.347276    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\""
Nov 01 21:00:00.349068 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.349043    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"]
Nov 01 21:00:00.385882 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.385854    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkfx\" (UniqueName: \"kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx\") pod \"maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.418955 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.418925    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qdh\" (UniqueName: \"kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.419118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419030    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2hd\" (UniqueName: \"kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.419118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419061    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.419118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419113    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.419306 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419192    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhcg\" (UniqueName: \"kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.419306 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419226    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.419756 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419735    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.419800 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419768    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.419865 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.419850    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.476379 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.476309    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qdh\" (UniqueName: \"kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.476379 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.476342    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhcg\" (UniqueName: \"kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.476578 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.476457    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2hd\" (UniqueName: \"kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.515444 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.515421    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:00.542223 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.542188    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:00.557871 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.557844    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:00.562556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.562537    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:00.902098 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.902075    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"]
Nov 01 21:00:00.905163 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:00:00.905126    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3989b761_01dc_4efc_9570_679548d75d21.slice/crio-0d7f9eedc348de6e9b3fa480dfbab7e7a535d9a007a3e69e66f955fc043c566d WatchSource:0}: Error finding container 0d7f9eedc348de6e9b3fa480dfbab7e7a535d9a007a3e69e66f955fc043c566d: Status 404 returned error can't find the container with id 0d7f9eedc348de6e9b3fa480dfbab7e7a535d9a007a3e69e66f955fc043c566d
Nov 01 21:00:00.954424 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:00.954402    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"]
Nov 01 21:00:00.956907 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:00:00.956879    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf161503e_529a_420e_b4d7_f20372c59a7a.slice/crio-c2f9cadda7edf43d9220b5a604dd086550beb433ce940b61763a077c2eeb793e WatchSource:0}: Error finding container c2f9cadda7edf43d9220b5a604dd086550beb433ce940b61763a077c2eeb793e: Status 404 returned error can't find the container with id c2f9cadda7edf43d9220b5a604dd086550beb433ce940b61763a077c2eeb793e
Nov 01 21:00:01.010597 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:01.008513    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"]
Nov 01 21:00:01.014956 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:01.014799    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"]
Nov 01 21:00:01.017998 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:00:01.016838    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e35d60_b06f_4819_b15f_b9ade46dc5b9.slice/crio-6d4ddd56a5d0dcb4376939afcfdd91a456f4313ac740803c0b67fb11b82d7c82 WatchSource:0}: Error finding container 6d4ddd56a5d0dcb4376939afcfdd91a456f4313ac740803c0b67fb11b82d7c82: Status 404 returned error can't find the container with id 6d4ddd56a5d0dcb4376939afcfdd91a456f4313ac740803c0b67fb11b82d7c82
Nov 01 21:00:01.444717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:01.444682    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerStarted","Data":"e5f36676a9fd31e85c8ec8edbafee82901e8a1a1979a52b83f68fdf14be5e701"}
Nov 01 21:00:01.446079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:01.446042    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerStarted","Data":"c2f9cadda7edf43d9220b5a604dd086550beb433ce940b61763a077c2eeb793e"}
Nov 01 21:00:01.447360 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:01.447336    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerStarted","Data":"0d7f9eedc348de6e9b3fa480dfbab7e7a535d9a007a3e69e66f955fc043c566d"}
Nov 01 21:00:01.448569 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:01.448546    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerStarted","Data":"6d4ddd56a5d0dcb4376939afcfdd91a456f4313ac740803c0b67fb11b82d7c82"}
Nov 01 21:00:03.772087 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.772059    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 21:00:03.892968 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.892940    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893123 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893077    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893191 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893139    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893274 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893248    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893277    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:03.893333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893288    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893437 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893397    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893437 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893405    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:03.893534 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893435    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893534 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893508    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893672 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893562    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket\") pod \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\" (UID: \"854f9bb0-0165-4bcd-8e45-fb91cf705f34\") "
Nov 01 21:00:03.893672 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893612    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:03.893832 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893802    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:03.893913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.893889    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:03.894509 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.894326    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.894509 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.894356    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.894509 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.894371    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.894509 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.894385    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.894509 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.894400    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.895896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.895864    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:03.895983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.895877    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:00:03.895983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.895930    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:03.896050 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.895994    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8" pod "854f9bb0-0165-4bcd-8e45-fb91cf705f34" (UID: "854f9bb0-0165-4bcd-8e45-fb91cf705f34"). InnerVolumeSpecName "kube-api-access-5ltw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:03.996726 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.996646    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.996726 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.996732    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ltw8\" (UniqueName: \"kubernetes.io/projected/854f9bb0-0165-4bcd-8e45-fb91cf705f34-kube-api-access-5ltw8\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.996970 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.996748    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:03.996970 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:03.996761    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854f9bb0-0165-4bcd-8e45-fb91cf705f34-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.477050 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.477003    4872 generic.go:356] "Generic (PLEG): container finished" podID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerID="8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466" exitCode=0
Nov 01 21:00:04.477235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.477068    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"
Nov 01 21:00:04.477235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.477088    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" event={"ID":"854f9bb0-0165-4bcd-8e45-fb91cf705f34","Type":"ContainerDied","Data":"8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466"}
Nov 01 21:00:04.477235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.477185    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj" event={"ID":"854f9bb0-0165-4bcd-8e45-fb91cf705f34","Type":"ContainerDied","Data":"0c94e2c2d10c6a32850840f08d870f19aa62d0b33b412e8179263cda3b89bdd4"}
Nov 01 21:00:04.477235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.477216    4872 scope.go:122] "RemoveContainer" containerID="8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466"
Nov 01 21:00:04.479924 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.479901    4872 generic.go:356] "Generic (PLEG): container finished" podID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerID="f64ce4e70706638491d92cf3ab7f1e9e51128bbac27040423a873c4f42362cb1" exitCode=0
Nov 01 21:00:04.480058 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.479968    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" event={"ID":"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03","Type":"ContainerDied","Data":"f64ce4e70706638491d92cf3ab7f1e9e51128bbac27040423a873c4f42362cb1"}
Nov 01 21:00:04.489414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.489395    4872 scope.go:122] "RemoveContainer" containerID="8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466"
Nov 01 21:00:04.489717 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:04.489697    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466\": container with ID starting with 8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466 not found: ID does not exist" containerID="8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466"
Nov 01 21:00:04.489780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.489727    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466"} err="failed to get container status \"8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466\": rpc error: code = NotFound desc = could not find container \"8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466\": container with ID starting with 8bf2fce352dcfb491633b168e593061ee878557fdab9e2ca1301da90a109d466 not found: ID does not exist"
Nov 01 21:00:04.532738 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.532719    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 21:00:04.606893 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.606866    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607062 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607045    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607104 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607077    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607300 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607279    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607300 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607294    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:04.607468 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607315    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607468 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607442    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607504    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:04.607556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607550    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607648 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607582    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.607648 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.607641    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo\") pod \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\" (UID: \"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03\") "
Nov 01 21:00:04.608134 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608005    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:04.608301 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608269    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:04.608697 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608675    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.608804 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608703    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.608804 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608717    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.608804 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608712    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:04.608804 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.608730    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.609450 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.609434    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:04.609883 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.609866    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "kube-api-access-zvkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:04.609944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.609889    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:00:04.610204 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.610189    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token" pod "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" (UID: "fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:04.648259 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.648232    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"]
Nov 01 21:00:04.698169 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.698135    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-fr9fj"]
Nov 01 21:00:04.710874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.710844    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkxf\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-kube-api-access-zvkxf\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.710874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.710876    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.711058 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.710890    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.711058 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.710902    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.711058 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.710914    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:04.859330 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:04.859308    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 21:00:05.015831 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.015789    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.016645 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.015882    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.016645 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.015980    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.016645 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.016346    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.016645 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.016580    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.017112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.016708    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.017112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.016739    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.017112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.016775    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:05.017112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.016815    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.017334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.017222    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.017386 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.017346    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.017436 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.017393    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.017498 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.017431    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo\") pod \"61510c86-9623-4b53-8682-da5f35e720a2\" (UID: \"61510c86-9623-4b53-8682-da5f35e720a2\") "
Nov 01 21:00:05.018183 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.018154    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.018280 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.018185    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.018280 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.018202    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61510c86-9623-4b53-8682-da5f35e720a2-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.018280 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.018216    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.019034 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.019007    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.021311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.020968    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.021837 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.021814    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:05.021935 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.021843    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "kube-api-access-7s6b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:05.022732 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.022705    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo" pod "61510c86-9623-4b53-8682-da5f35e720a2" (UID: "61510c86-9623-4b53-8682-da5f35e720a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:00:05.045996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.045969    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 21:00:05.046165 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.046024    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 21:00:05.087753 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.087709    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" path="/var/lib/kubelet/pods/854f9bb0-0165-4bcd-8e45-fb91cf705f34/volumes"
Nov 01 21:00:05.122541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.121472    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.122541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.121513    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s6b7\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-kube-api-access-7s6b7\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.122541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.121533    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61510c86-9623-4b53-8682-da5f35e720a2-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.122541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.121546    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61510c86-9623-4b53-8682-da5f35e720a2-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.122541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.121557    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61510c86-9623-4b53-8682-da5f35e720a2-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.234212 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.234183    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 21:00:05.324545 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.324509    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.324761 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.324559    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.324761 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.324593    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.324901 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.324877    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.325404 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.325362    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:05.325568 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.325545    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.325706 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.325582    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.325824 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.325736    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.325909 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.325862    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.325979 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.325938    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.326115 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.326015    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data\") pod \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\" (UID: \"ffc63ce5-6af2-4200-a4c0-86ffe970aab9\") "
Nov 01 21:00:05.326471 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.326442    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.326679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.326630    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.327894 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.327870    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.327998 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.327900    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.327998 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.327914    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.327998 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.327929    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.328153 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.328034    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:00:05.328153 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.328087    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.330616 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.330591    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "kube-api-access-m4qgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:05.330743 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.330613    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:00:05.332073 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.332047    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token" pod "ffc63ce5-6af2-4200-a4c0-86ffe970aab9" (UID: "ffc63ce5-6af2-4200-a4c0-86ffe970aab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:05.430203 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.430168    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.430203 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.430206    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.430446 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.430221    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.430446 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.430235    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.430446 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.430249    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qgj\" (UniqueName: \"kubernetes.io/projected/ffc63ce5-6af2-4200-a4c0-86ffe970aab9-kube-api-access-m4qgj\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:05.496586 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.496548    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4" event={"ID":"fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03","Type":"ContainerDied","Data":"45934766e56674213430ef8ae4ee1eca681de360b54d97a4c5856edb7e3fd7ef"}
Nov 01 21:00:05.496796 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.496624    4872 scope.go:122] "RemoveContainer" containerID="f64ce4e70706638491d92cf3ab7f1e9e51128bbac27040423a873c4f42362cb1"
Nov 01 21:00:05.496796 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.496746    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"
Nov 01 21:00:05.500923 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.500893    4872 generic.go:356] "Generic (PLEG): container finished" podID="61510c86-9623-4b53-8682-da5f35e720a2" containerID="ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934" exitCode=0
Nov 01 21:00:05.501088 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.501066    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" event={"ID":"61510c86-9623-4b53-8682-da5f35e720a2","Type":"ContainerDied","Data":"ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934"}
Nov 01 21:00:05.501178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.501118    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v" event={"ID":"61510c86-9623-4b53-8682-da5f35e720a2","Type":"ContainerDied","Data":"026d9eeb79bc172616b52a5805b0750a80333d0a03324604926e33622d475cf8"}
Nov 01 21:00:05.501239 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.501215    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"
Nov 01 21:00:05.507186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.507144    4872 generic.go:356] "Generic (PLEG): container finished" podID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerID="8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb" exitCode=0
Nov 01 21:00:05.507327 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.507315    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"
Nov 01 21:00:05.507811 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.507792    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" event={"ID":"ffc63ce5-6af2-4200-a4c0-86ffe970aab9","Type":"ContainerDied","Data":"8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb"}
Nov 01 21:00:05.507901 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.507844    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd" event={"ID":"ffc63ce5-6af2-4200-a4c0-86ffe970aab9","Type":"ContainerDied","Data":"1b2c90a819c8bfed1c74df782f88d488ddcdfc93170c13b82ff2489f6fb65585"}
Nov 01 21:00:05.540763 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.540573    4872 scope.go:122] "RemoveContainer" containerID="ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934"
Nov 01 21:00:05.572032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.572014    4872 scope.go:122] "RemoveContainer" containerID="ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934"
Nov 01 21:00:05.572421 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:05.572401    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934\": container with ID starting with ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934 not found: ID does not exist" containerID="ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934"
Nov 01 21:00:05.572502 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.572434    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934"} err="failed to get container status \"ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934\": rpc error: code = NotFound desc = could not find container \"ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934\": container with ID starting with ffece15664f2002a543605c406c33e69491688a81d6a7e3c4e1081cf4069f934 not found: ID does not exist"
Nov 01 21:00:05.572502 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.572454    4872 scope.go:122] "RemoveContainer" containerID="8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb"
Nov 01 21:00:05.584883 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.584859    4872 scope.go:122] "RemoveContainer" containerID="8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb"
Nov 01 21:00:05.585304 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:05.585246    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb\": container with ID starting with 8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb not found: ID does not exist" containerID="8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb"
Nov 01 21:00:05.585304 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.585278    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb"} err="failed to get container status \"8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb\": rpc error: code = NotFound desc = could not find container \"8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb\": container with ID starting with 8b599674b067b4185eb81f5ddb2a0f401bd111ada285df78c882db42d92726bb not found: ID does not exist"
Nov 01 21:00:05.677417 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.677386    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"]
Nov 01 21:00:05.685406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.685329    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-65cf4"]
Nov 01 21:00:05.797016 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.796961    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"]
Nov 01 21:00:05.801944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.801896    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-b9n9v"]
Nov 01 21:00:05.913477 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.913447    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"]
Nov 01 21:00:05.920778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:05.920753    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-hnrhd"]
Nov 01 21:00:07.085515 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:07.085481    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61510c86-9623-4b53-8682-da5f35e720a2" path="/var/lib/kubelet/pods/61510c86-9623-4b53-8682-da5f35e720a2/volumes"
Nov 01 21:00:07.086210 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:07.086186    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" path="/var/lib/kubelet/pods/fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03/volumes"
Nov 01 21:00:07.086926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:07.086904    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" path="/var/lib/kubelet/pods/ffc63ce5-6af2-4200-a4c0-86ffe970aab9/volumes"
Nov 01 21:00:08.548864 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:08.548828    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerStarted","Data":"f772e688ca64096630024f14d0562d78b27d84b58b9f4392c458a5ab3a2576b0"}
Nov 01 21:00:08.551147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:08.551116    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerStarted","Data":"c39fc65dcc6be30388b51c9f5f20ad3bb12f8ee5ac068db319cbb82ed3a63b2b"}
Nov 01 21:00:08.553536 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:08.553503    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerStarted","Data":"a6451fb37a10b6d0af92750411c4acb1dbdc53b7ea8a5c746a4b2ffbb5e1a980"}
Nov 01 21:00:08.665304 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:08.665251    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" podStartSLOduration=1.7786826740000001 podStartE2EDuration="8.665237236s" totalImagesPullingTime="6.886554562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:08.664376129 +0000 UTC m=+1765.713485946" watchObservedRunningTime="2026-11-01 21:00:08.665237236 +0000 UTC m=+1765.714347053"
Nov 01 21:00:08.784275 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:08.784190    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" podStartSLOduration=1.825448924 podStartE2EDuration="8.784174134s" totalImagesPullingTime="6.95872521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:08.779541402 +0000 UTC m=+1765.828651220" watchObservedRunningTime="2026-11-01 21:00:08.784174134 +0000 UTC m=+1765.833283953"
Nov 01 21:00:08.899594 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:08.899527    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" podStartSLOduration=1.911813831 podStartE2EDuration="8.89950826s" totalImagesPullingTime="6.987694429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:08.896836589 +0000 UTC m=+1765.945946419" watchObservedRunningTime="2026-11-01 21:00:08.89950826 +0000 UTC m=+1765.948618079"
Nov 01 21:00:09.563190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.563158    4872 generic.go:356] "Generic (PLEG): container finished" podID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerID="a6451fb37a10b6d0af92750411c4acb1dbdc53b7ea8a5c746a4b2ffbb5e1a980" exitCode=60
Nov 01 21:00:09.563576 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.563236    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerDied","Data":"a6451fb37a10b6d0af92750411c4acb1dbdc53b7ea8a5c746a4b2ffbb5e1a980"}
Nov 01 21:00:09.563576 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.563439    4872 scope.go:122] "RemoveContainer" containerID="a6451fb37a10b6d0af92750411c4acb1dbdc53b7ea8a5c746a4b2ffbb5e1a980"
Nov 01 21:00:09.565003 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.564981    4872 generic.go:356] "Generic (PLEG): container finished" podID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerID="f772e688ca64096630024f14d0562d78b27d84b58b9f4392c458a5ab3a2576b0" exitCode=60
Nov 01 21:00:09.565127 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.565061    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerDied","Data":"f772e688ca64096630024f14d0562d78b27d84b58b9f4392c458a5ab3a2576b0"}
Nov 01 21:00:09.565281 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.565265    4872 scope.go:122] "RemoveContainer" containerID="f772e688ca64096630024f14d0562d78b27d84b58b9f4392c458a5ab3a2576b0"
Nov 01 21:00:09.566862 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.566837    4872 generic.go:356] "Generic (PLEG): container finished" podID="f161503e-529a-420e-b4d7-f20372c59a7a" containerID="af92eb0bfba86a3789f454638d639270286bbe1a0f5e3b9dbec889a323a0cb81" exitCode=60
Nov 01 21:00:09.566940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.566913    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerDied","Data":"af92eb0bfba86a3789f454638d639270286bbe1a0f5e3b9dbec889a323a0cb81"}
Nov 01 21:00:09.567133 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.567117    4872 scope.go:122] "RemoveContainer" containerID="af92eb0bfba86a3789f454638d639270286bbe1a0f5e3b9dbec889a323a0cb81"
Nov 01 21:00:09.570141 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.568432    4872 generic.go:356] "Generic (PLEG): container finished" podID="3989b761-01dc-4efc-9570-679548d75d21" containerID="c39fc65dcc6be30388b51c9f5f20ad3bb12f8ee5ac068db319cbb82ed3a63b2b" exitCode=60
Nov 01 21:00:09.570141 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.568497    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerDied","Data":"c39fc65dcc6be30388b51c9f5f20ad3bb12f8ee5ac068db319cbb82ed3a63b2b"}
Nov 01 21:00:09.570141 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:09.568705    4872 scope.go:122] "RemoveContainer" containerID="c39fc65dcc6be30388b51c9f5f20ad3bb12f8ee5ac068db319cbb82ed3a63b2b"
Nov 01 21:00:10.578013 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.577979    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerStarted","Data":"e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294"}
Nov 01 21:00:10.580014 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.579990    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerStarted","Data":"e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76"}
Nov 01 21:00:10.581785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.581761    4872 generic.go:356] "Generic (PLEG): container finished" podID="3989b761-01dc-4efc-9570-679548d75d21" containerID="532028ed33f580e9e5556364b7f2245fcf651e7c83136996f70361bb08ea7b9c" exitCode=60
Nov 01 21:00:10.581918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.581835    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerDied","Data":"532028ed33f580e9e5556364b7f2245fcf651e7c83136996f70361bb08ea7b9c"}
Nov 01 21:00:10.581918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.581888    4872 scope.go:122] "RemoveContainer" containerID="c39fc65dcc6be30388b51c9f5f20ad3bb12f8ee5ac068db319cbb82ed3a63b2b"
Nov 01 21:00:10.582049 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.582035    4872 scope.go:122] "RemoveContainer" containerID="532028ed33f580e9e5556364b7f2245fcf651e7c83136996f70361bb08ea7b9c"
Nov 01 21:00:10.582337 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:10.582316    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd_odh-ai-gateway-infra(3989b761-01dc-4efc-9570-679548d75d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" podUID="3989b761-01dc-4efc-9570-679548d75d21"
Nov 01 21:00:10.583906 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.583887    4872 generic.go:356] "Generic (PLEG): container finished" podID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerID="cba3613b949298c19d589eb1154a9b233a38402884d5d061161c5de3cb25acd9" exitCode=60
Nov 01 21:00:10.584011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.583914    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerDied","Data":"cba3613b949298c19d589eb1154a9b233a38402884d5d061161c5de3cb25acd9"}
Nov 01 21:00:10.584140 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.584127    4872 scope.go:122] "RemoveContainer" containerID="cba3613b949298c19d589eb1154a9b233a38402884d5d061161c5de3cb25acd9"
Nov 01 21:00:10.584370 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:10.584352    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn_odh-ai-gateway-infra(12e35d60-b06f-4819-b15f-b9ade46dc5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9"
Nov 01 21:00:10.594299 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.594282    4872 scope.go:122] "RemoveContainer" containerID="a6451fb37a10b6d0af92750411c4acb1dbdc53b7ea8a5c746a4b2ffbb5e1a980"
Nov 01 21:00:10.952505 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:10.952480    4872 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]"
Nov 01 21:00:10.963111 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:10.963064    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" podStartSLOduration=3.680886885 podStartE2EDuration="10.963051248s" totalImagesPullingTime="7.282164363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:10.82659819 +0000 UTC m=+1767.875708009" watchObservedRunningTime="2026-11-01 21:00:10.963051248 +0000 UTC m=+1768.012161107"
Nov 01 21:00:11.593533 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.593503    4872 generic.go:356] "Generic (PLEG): container finished" podID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerID="e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294" exitCode=60
Nov 01 21:00:11.594016 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.593590    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerDied","Data":"e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294"}
Nov 01 21:00:11.594016 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.593696    4872 scope.go:122] "RemoveContainer" containerID="f772e688ca64096630024f14d0562d78b27d84b58b9f4392c458a5ab3a2576b0"
Nov 01 21:00:11.594016 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.593825    4872 scope.go:122] "RemoveContainer" containerID="e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294"
Nov 01 21:00:11.594164 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:11.594141    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d_odh-ai-gateway-infra(f2a483e8-0178-4c2b-ae88-ab55b5f9ec61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61"
Nov 01 21:00:11.595714 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.595690    4872 generic.go:356] "Generic (PLEG): container finished" podID="f161503e-529a-420e-b4d7-f20372c59a7a" containerID="e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76" exitCode=60
Nov 01 21:00:11.595842 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.595767    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerDied","Data":"e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76"}
Nov 01 21:00:11.596257 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.595971    4872 scope.go:122] "RemoveContainer" containerID="e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76"
Nov 01 21:00:11.596257 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:11.596206    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8_odh-ai-gateway-infra(f161503e-529a-420e-b4d7-f20372c59a7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" podUID="f161503e-529a-420e-b4d7-f20372c59a7a"
Nov 01 21:00:11.597457 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.597441    4872 scope.go:122] "RemoveContainer" containerID="532028ed33f580e9e5556364b7f2245fcf651e7c83136996f70361bb08ea7b9c"
Nov 01 21:00:11.597688 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:11.597667    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd_odh-ai-gateway-infra(3989b761-01dc-4efc-9570-679548d75d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" podUID="3989b761-01dc-4efc-9570-679548d75d21"
Nov 01 21:00:11.598999 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.598983    4872 scope.go:122] "RemoveContainer" containerID="cba3613b949298c19d589eb1154a9b233a38402884d5d061161c5de3cb25acd9"
Nov 01 21:00:11.599208 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:11.599186    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn_odh-ai-gateway-infra(12e35d60-b06f-4819-b15f-b9ade46dc5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9"
Nov 01 21:00:11.607731 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:11.607713    4872 scope.go:122] "RemoveContainer" containerID="af92eb0bfba86a3789f454638d639270286bbe1a0f5e3b9dbec889a323a0cb81"
Nov 01 21:00:12.607910 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:12.607879    4872 scope.go:122] "RemoveContainer" containerID="e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294"
Nov 01 21:00:12.608381 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:12.608206    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d_odh-ai-gateway-infra(f2a483e8-0178-4c2b-ae88-ab55b5f9ec61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61"
Nov 01 21:00:12.610251 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:12.609897    4872 scope.go:122] "RemoveContainer" containerID="e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76"
Nov 01 21:00:12.610251 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:12.610199    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8_odh-ai-gateway-infra(f161503e-529a-420e-b4d7-f20372c59a7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" podUID="f161503e-529a-420e-b4d7-f20372c59a7a"
Nov 01 21:00:15.572070 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:15.572036    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"]
Nov 01 21:00:15.654404 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:15.654374    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"]
Nov 01 21:00:20.572556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:20.572500    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" podUID="daee4916-972b-4f58-a7b5-9027dd759157" containerName="payload-processing" containerID="cri-o://14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a" gracePeriod=25
Nov 01 21:00:20.655213 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:20.655171    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" podUID="28b767ec-a136-4b44-b740-79f223592863" containerName="payload-pre-processing" containerID="cri-o://ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b" gracePeriod=25
Nov 01 21:00:21.081964 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:21.081938    4872 scope.go:122] "RemoveContainer" containerID="e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294"
Nov 01 21:00:21.082146 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:21.082079    4872 scope.go:122] "RemoveContainer" containerID="e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76"
Nov 01 21:00:21.707237 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:21.707204    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerStarted","Data":"c71a8cb5ca0a05fee732f1adb564150a18cf627ec739e33d7f5d680629821e24"}
Nov 01 21:00:21.709359 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:21.709328    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerStarted","Data":"1d228a2de7767588099fca1a210cce4b58501366137e0c5303a6fa61c49544a5"}
Nov 01 21:00:22.717912 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.717879    4872 generic.go:356] "Generic (PLEG): container finished" podID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerID="c71a8cb5ca0a05fee732f1adb564150a18cf627ec739e33d7f5d680629821e24" exitCode=60
Nov 01 21:00:22.718314 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.717946    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerDied","Data":"c71a8cb5ca0a05fee732f1adb564150a18cf627ec739e33d7f5d680629821e24"}
Nov 01 21:00:22.718314 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.718025    4872 scope.go:122] "RemoveContainer" containerID="e35af4fe446327e4373a3c65a3357fb3f6e2c78fc2c2f879698ebb4809bcf294"
Nov 01 21:00:22.718436 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.718351    4872 scope.go:122] "RemoveContainer" containerID="c71a8cb5ca0a05fee732f1adb564150a18cf627ec739e33d7f5d680629821e24"
Nov 01 21:00:22.718712 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:22.718684    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d_odh-ai-gateway-infra(f2a483e8-0178-4c2b-ae88-ab55b5f9ec61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61"
Nov 01 21:00:22.719978 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.719954    4872 generic.go:356] "Generic (PLEG): container finished" podID="f161503e-529a-420e-b4d7-f20372c59a7a" containerID="1d228a2de7767588099fca1a210cce4b58501366137e0c5303a6fa61c49544a5" exitCode=60
Nov 01 21:00:22.720055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.720020    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerDied","Data":"1d228a2de7767588099fca1a210cce4b58501366137e0c5303a6fa61c49544a5"}
Nov 01 21:00:22.720325 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.720312    4872 scope.go:122] "RemoveContainer" containerID="1d228a2de7767588099fca1a210cce4b58501366137e0c5303a6fa61c49544a5"
Nov 01 21:00:22.720566 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:22.720550    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8_odh-ai-gateway-infra(f161503e-529a-420e-b4d7-f20372c59a7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" podUID="f161503e-529a-420e-b4d7-f20372c59a7a"
Nov 01 21:00:22.730719 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.730702    4872 scope.go:122] "RemoveContainer" containerID="e8444dad3252e9e7833613fe6af674557dd7e018a42f3d9ec7dd566be876bb76"
Nov 01 21:00:22.833937 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.833911    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"]
Nov 01 21:00:22.953690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:22.953626    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"]
Nov 01 21:00:23.368703 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:23.368647    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:00:23.910720 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:23.910699    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:23.950821 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:23.950795    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:24.095778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.095693    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd\" (UniqueName: \"kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd\") pod \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") "
Nov 01 21:00:24.095778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.095761    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca\") pod \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\" (UID: \"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61\") "
Nov 01 21:00:24.096004 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.095961    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg\" (UniqueName: \"kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg\") pod \"f161503e-529a-420e-b4d7-f20372c59a7a\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") "
Nov 01 21:00:24.096069 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.096054    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca\") pod \"f161503e-529a-420e-b4d7-f20372c59a7a\" (UID: \"f161503e-529a-420e-b4d7-f20372c59a7a\") "
Nov 01 21:00:24.097021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.096995    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca" pod "f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" (UID: "f2a483e8-0178-4c2b-ae88-ab55b5f9ec61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:24.097276 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.097256    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca" pod "f161503e-529a-420e-b4d7-f20372c59a7a" (UID: "f161503e-529a-420e-b4d7-f20372c59a7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:24.099174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.099148    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd" pod "f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" (UID: "f2a483e8-0178-4c2b-ae88-ab55b5f9ec61"). InnerVolumeSpecName "kube-api-access-tr2hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:24.099399 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.099372    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg" pod "f161503e-529a-420e-b4d7-f20372c59a7a" (UID: "f161503e-529a-420e-b4d7-f20372c59a7a"). InnerVolumeSpecName "kube-api-access-kkhcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:24.198391 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.198363    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr2hd\" (UniqueName: \"kubernetes.io/projected/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-kube-api-access-tr2hd\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:24.198391 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.198385    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61-maas-api-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:24.198391 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.198396    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhcg\" (UniqueName: \"kubernetes.io/projected/f161503e-529a-420e-b4d7-f20372c59a7a-kube-api-access-kkhcg\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:24.198587 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.198404    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f161503e-529a-420e-b4d7-f20372c59a7a-maas-api-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:24.738345 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.738315    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"
Nov 01 21:00:24.738345 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.738333    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8" event={"ID":"f161503e-529a-420e-b4d7-f20372c59a7a","Type":"ContainerDied","Data":"c2f9cadda7edf43d9220b5a604dd086550beb433ce940b61763a077c2eeb793e"}
Nov 01 21:00:24.738736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.738417    4872 scope.go:122] "RemoveContainer" containerID="1d228a2de7767588099fca1a210cce4b58501366137e0c5303a6fa61c49544a5"
Nov 01 21:00:24.740065 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.740042    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d" event={"ID":"f2a483e8-0178-4c2b-ae88-ab55b5f9ec61","Type":"ContainerDied","Data":"e5f36676a9fd31e85c8ec8edbafee82901e8a1a1979a52b83f68fdf14be5e701"}
Nov 01 21:00:24.740130 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.740095    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"
Nov 01 21:00:24.749332 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.749312    4872 scope.go:122] "RemoveContainer" containerID="c71a8cb5ca0a05fee732f1adb564150a18cf627ec739e33d7f5d680629821e24"
Nov 01 21:00:24.907315 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.907287    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"]
Nov 01 21:00:24.914477 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:24.914451    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-de8145-29892780-j6h8d"]
Nov 01 21:00:25.024534 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.024465    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"]
Nov 01 21:00:25.027851 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.027827    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892780-xpvm8"]
Nov 01 21:00:25.080192 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.080165    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" path="/var/lib/kubelet/pods/f161503e-529a-420e-b4d7-f20372c59a7a/volumes"
Nov 01 21:00:25.080648 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.080634    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" path="/var/lib/kubelet/pods/f2a483e8-0178-4c2b-ae88-ab55b5f9ec61/volumes"
Nov 01 21:00:25.081484 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.081459    4872 scope.go:122] "RemoveContainer" containerID="cba3613b949298c19d589eb1154a9b233a38402884d5d061161c5de3cb25acd9"
Nov 01 21:00:25.081779 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.081761    4872 scope.go:122] "RemoveContainer" containerID="532028ed33f580e9e5556364b7f2245fcf651e7c83136996f70361bb08ea7b9c"
Nov 01 21:00:25.357612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.357578    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" podUID="28b767ec-a136-4b44-b740-79f223592863" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:25.357786 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.357610    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" podUID="daee4916-972b-4f58-a7b5-9027dd759157" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:25.751251 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.751220    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerStarted","Data":"754035911c81a46c3fd2c9ba13001c7edee39431f2fc84265397d26ef57d6a1b"}
Nov 01 21:00:25.753157 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:25.753131    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerStarted","Data":"83d9d0f7648e862e6558025cb3f87c72c132b501e4c5035bc013f47d014f4cef"}
Nov 01 21:00:26.807445 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.807412    4872 generic.go:356] "Generic (PLEG): container finished" podID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerID="83d9d0f7648e862e6558025cb3f87c72c132b501e4c5035bc013f47d014f4cef" exitCode=60
Nov 01 21:00:26.807940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.807641    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerDied","Data":"83d9d0f7648e862e6558025cb3f87c72c132b501e4c5035bc013f47d014f4cef"}
Nov 01 21:00:26.807940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.807719    4872 scope.go:122] "RemoveContainer" containerID="cba3613b949298c19d589eb1154a9b233a38402884d5d061161c5de3cb25acd9"
Nov 01 21:00:26.812396 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.812365    4872 scope.go:122] "RemoveContainer" containerID="83d9d0f7648e862e6558025cb3f87c72c132b501e4c5035bc013f47d014f4cef"
Nov 01 21:00:26.812867 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:26.812827    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn_odh-ai-gateway-infra(12e35d60-b06f-4819-b15f-b9ade46dc5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9"
Nov 01 21:00:26.814179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.814147    4872 generic.go:356] "Generic (PLEG): container finished" podID="3989b761-01dc-4efc-9570-679548d75d21" containerID="754035911c81a46c3fd2c9ba13001c7edee39431f2fc84265397d26ef57d6a1b" exitCode=60
Nov 01 21:00:26.814309 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.814263    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerDied","Data":"754035911c81a46c3fd2c9ba13001c7edee39431f2fc84265397d26ef57d6a1b"}
Nov 01 21:00:26.814730 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.814711    4872 scope.go:122] "RemoveContainer" containerID="754035911c81a46c3fd2c9ba13001c7edee39431f2fc84265397d26ef57d6a1b"
Nov 01 21:00:26.815138 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:26.815112    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd_odh-ai-gateway-infra(3989b761-01dc-4efc-9570-679548d75d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" podUID="3989b761-01dc-4efc-9570-679548d75d21"
Nov 01 21:00:26.822877 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.822861    4872 scope.go:122] "RemoveContainer" containerID="532028ed33f580e9e5556364b7f2245fcf651e7c83136996f70361bb08ea7b9c"
Nov 01 21:00:26.915467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.915430    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"]
Nov 01 21:00:26.995556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:26.995517    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"]
Nov 01 21:00:28.019384 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.019351    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:28.058283 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.058248    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:28.073435 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.073392    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca\") pod \"3989b761-01dc-4efc-9570-679548d75d21\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") "
Nov 01 21:00:28.073573 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.073556    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh\" (UniqueName: \"kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh\") pod \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") "
Nov 01 21:00:28.073649 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.073583    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx\" (UniqueName: \"kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx\") pod \"3989b761-01dc-4efc-9570-679548d75d21\" (UID: \"3989b761-01dc-4efc-9570-679548d75d21\") "
Nov 01 21:00:28.073747 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.073678    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca\") pod \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\" (UID: \"12e35d60-b06f-4819-b15f-b9ade46dc5b9\") "
Nov 01 21:00:28.073807 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.073767    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca" pod "3989b761-01dc-4efc-9570-679548d75d21" (UID: "3989b761-01dc-4efc-9570-679548d75d21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:28.074016 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.073993    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca" pod "12e35d60-b06f-4819-b15f-b9ade46dc5b9" (UID: "12e35d60-b06f-4819-b15f-b9ade46dc5b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:28.074323 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.074298    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e35d60-b06f-4819-b15f-b9ade46dc5b9-maas-api-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:28.074323 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.074319    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3989b761-01dc-4efc-9570-679548d75d21-maas-api-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:28.075761 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.075735    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh" pod "12e35d60-b06f-4819-b15f-b9ade46dc5b9" (UID: "12e35d60-b06f-4819-b15f-b9ade46dc5b9"). InnerVolumeSpecName "kube-api-access-d6qdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:28.075889 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.075867    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx" pod "3989b761-01dc-4efc-9570-679548d75d21" (UID: "3989b761-01dc-4efc-9570-679548d75d21"). InnerVolumeSpecName "kube-api-access-2fkfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:28.178506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.178316    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qdh\" (UniqueName: \"kubernetes.io/projected/12e35d60-b06f-4819-b15f-b9ade46dc5b9-kube-api-access-d6qdh\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:28.178506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.178350    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkfx\" (UniqueName: \"kubernetes.io/projected/3989b761-01dc-4efc-9570-679548d75d21-kube-api-access-2fkfx\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:28.840454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.838673    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"
Nov 01 21:00:28.840454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.838696    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd" event={"ID":"3989b761-01dc-4efc-9570-679548d75d21","Type":"ContainerDied","Data":"0d7f9eedc348de6e9b3fa480dfbab7e7a535d9a007a3e69e66f955fc043c566d"}
Nov 01 21:00:28.840454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.838801    4872 scope.go:122] "RemoveContainer" containerID="754035911c81a46c3fd2c9ba13001c7edee39431f2fc84265397d26ef57d6a1b"
Nov 01 21:00:28.841324 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.841227    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn" event={"ID":"12e35d60-b06f-4819-b15f-b9ade46dc5b9","Type":"ContainerDied","Data":"6d4ddd56a5d0dcb4376939afcfdd91a456f4313ac740803c0b67fb11b82d7c82"}
Nov 01 21:00:28.841324 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.841308    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"
Nov 01 21:00:28.853891 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:28.853873    4872 scope.go:122] "RemoveContainer" containerID="83d9d0f7648e862e6558025cb3f87c72c132b501e4c5035bc013f47d014f4cef"
Nov 01 21:00:29.007844 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:29.007806    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"]
Nov 01 21:00:29.056214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:29.056186    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892780-bvntn"]
Nov 01 21:00:29.080722 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:29.080695    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" path="/var/lib/kubelet/pods/12e35d60-b06f-4819-b15f-b9ade46dc5b9/volumes"
Nov 01 21:00:29.122305 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:29.121745    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"]
Nov 01 21:00:29.125805 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:29.125780    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11a069b1-29892780-w2jxd"]
Nov 01 21:00:31.081145 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:31.081114    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3989b761-01dc-4efc-9570-679548d75d21" path="/var/lib/kubelet/pods/3989b761-01dc-4efc-9570-679548d75d21/volumes"
Nov 01 21:00:35.357595 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:35.357555    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" podUID="28b767ec-a136-4b44-b740-79f223592863" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:35.358085 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:35.357565    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" podUID="daee4916-972b-4f58-a7b5-9027dd759157" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:40.881596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.881570    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 21:00:40.907566 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.907544    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 21:00:40.936486 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.936456    4872 generic.go:356] "Generic (PLEG): container finished" podID="daee4916-972b-4f58-a7b5-9027dd759157" containerID="14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a" exitCode=0
Nov 01 21:00:40.936628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.936491    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"
Nov 01 21:00:40.936628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.936529    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" event={"ID":"daee4916-972b-4f58-a7b5-9027dd759157","Type":"ContainerDied","Data":"14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a"}
Nov 01 21:00:40.936628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.936610    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x" event={"ID":"daee4916-972b-4f58-a7b5-9027dd759157","Type":"ContainerDied","Data":"c65e809f83116ff783d3ac40f625eca945d14086a75dd23c28c566cd97d8aea8"}
Nov 01 21:00:40.936628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.936625    4872 scope.go:122] "RemoveContainer" containerID="14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a"
Nov 01 21:00:40.938170 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.938150    4872 generic.go:356] "Generic (PLEG): container finished" podID="28b767ec-a136-4b44-b740-79f223592863" containerID="ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b" exitCode=0
Nov 01 21:00:40.938253 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.938178    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" event={"ID":"28b767ec-a136-4b44-b740-79f223592863","Type":"ContainerDied","Data":"ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b"}
Nov 01 21:00:40.938253 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.938206    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"
Nov 01 21:00:40.938253 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.938232    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl" event={"ID":"28b767ec-a136-4b44-b740-79f223592863","Type":"ContainerDied","Data":"0fe61ad9db5cf471fab8e63d7e1e72338e33cb194f7e30b37435bbe8ddbf3c00"}
Nov 01 21:00:40.947320 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.947291    4872 scope.go:122] "RemoveContainer" containerID="14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a"
Nov 01 21:00:40.947566 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:40.947547    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a\": container with ID starting with 14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a not found: ID does not exist" containerID="14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a"
Nov 01 21:00:40.947626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.947583    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a"} err="failed to get container status \"14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a\": rpc error: code = NotFound desc = could not find container \"14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a\": container with ID starting with 14ae2c977349b72b571f1ce9677ab8d02b148e011b8e0c2d7adc72ab29bf760a not found: ID does not exist"
Nov 01 21:00:40.947626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.947600    4872 scope.go:122] "RemoveContainer" containerID="ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b"
Nov 01 21:00:40.960042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.960026    4872 scope.go:122] "RemoveContainer" containerID="ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b"
Nov 01 21:00:40.960270 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:40.960250    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b\": container with ID starting with ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b not found: ID does not exist" containerID="ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b"
Nov 01 21:00:40.960315 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.960277    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b"} err="failed to get container status \"ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b\": rpc error: code = NotFound desc = could not find container \"ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b\": container with ID starting with ab798fb9ed7e29fc31e814d4633b7f49a58e5f9c70cf1e75359b2ce64cc5a44b not found: ID does not exist"
Nov 01 21:00:40.985916 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.985895    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume\") pod \"28b767ec-a136-4b44-b740-79f223592863\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") "
Nov 01 21:00:40.986172 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.986158    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv\" (UniqueName: \"kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv\") pod \"28b767ec-a136-4b44-b740-79f223592863\" (UID: \"28b767ec-a136-4b44-b740-79f223592863\") "
Nov 01 21:00:40.986240 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.986222    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume" pod "28b767ec-a136-4b44-b740-79f223592863" (UID: "28b767ec-a136-4b44-b740-79f223592863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:40.986597 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.986583    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b767ec-a136-4b44-b740-79f223592863-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:40.988518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:40.988488    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv" pod "28b767ec-a136-4b44-b740-79f223592863" (UID: "28b767ec-a136-4b44-b740-79f223592863"). InnerVolumeSpecName "kube-api-access-7gndv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:41.087215 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.087130    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume\") pod \"daee4916-972b-4f58-a7b5-9027dd759157\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") "
Nov 01 21:00:41.087376 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.087339    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf\" (UniqueName: \"kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf\") pod \"daee4916-972b-4f58-a7b5-9027dd759157\" (UID: \"daee4916-972b-4f58-a7b5-9027dd759157\") "
Nov 01 21:00:41.087550 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.087525    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume" pod "daee4916-972b-4f58-a7b5-9027dd759157" (UID: "daee4916-972b-4f58-a7b5-9027dd759157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:41.087908 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.087878    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gndv\" (UniqueName: \"kubernetes.io/projected/28b767ec-a136-4b44-b740-79f223592863-kube-api-access-7gndv\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:41.087908 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.087902    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee4916-972b-4f58-a7b5-9027dd759157-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:41.089565 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.089538    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf" pod "daee4916-972b-4f58-a7b5-9027dd759157" (UID: "daee4916-972b-4f58-a7b5-9027dd759157"). InnerVolumeSpecName "kube-api-access-r6bwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:41.189211 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.189177    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6bwf\" (UniqueName: \"kubernetes.io/projected/daee4916-972b-4f58-a7b5-9027dd759157-kube-api-access-r6bwf\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:41.356136 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.356040    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"]
Nov 01 21:00:41.359450 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.359421    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-kh7wl"]
Nov 01 21:00:41.469710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.469682    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"]
Nov 01 21:00:41.472900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:41.472878    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-nqm4x"]
Nov 01 21:00:43.081785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:43.081753    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b767ec-a136-4b44-b740-79f223592863" path="/var/lib/kubelet/pods/28b767ec-a136-4b44-b740-79f223592863/volumes"
Nov 01 21:00:43.082214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:43.082199    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daee4916-972b-4f58-a7b5-9027dd759157" path="/var/lib/kubelet/pods/daee4916-972b-4f58-a7b5-9027dd759157/volumes"
Nov 01 21:00:43.440282 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:43.440247    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:00:43.441297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:43.441276    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:00:43.446050 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:43.446029    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:00:43.447214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:43.447193    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:00:53.366723 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:00:53.366689    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:00:54.154011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.153979    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"]
Nov 01 21:00:54.154609 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154595    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154611    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154622    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154627    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154639    4872 cpu_manager.go:414] "containerMap: removing container" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154644    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154664    4872 cpu_manager.go:414] "containerMap: removing container" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154670    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154679    4872 cpu_manager.go:414] "containerMap: removing container" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154683    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy"
Nov 01 21:00:54.154690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154691    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154696    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154703    4872 cpu_manager.go:414] "containerMap: removing container" podUID="28b767ec-a136-4b44-b740-79f223592863" containerName="payload-pre-processing"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154707    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b767ec-a136-4b44-b740-79f223592863" containerName="payload-pre-processing"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154719    4872 cpu_manager.go:414] "containerMap: removing container" podUID="daee4916-972b-4f58-a7b5-9027dd759157" containerName="payload-processing"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154723    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daee4916-972b-4f58-a7b5-9027dd759157" containerName="payload-processing"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154731    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154735    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154740    4872 cpu_manager.go:414] "containerMap: removing container" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154744    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154754    4872 cpu_manager.go:414] "containerMap: removing container" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154758    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154765    4872 cpu_manager.go:414] "containerMap: removing container" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154769    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154779    4872 cpu_manager.go:414] "containerMap: removing container" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154783    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154794    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154799    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154808    4872 cpu_manager.go:414] "containerMap: removing container" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154812    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154906    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb7e387-fb56-4ea9-b83f-a00bdeaa3c03" containerName="istio-proxy"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154916    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="61510c86-9623-4b53-8682-da5f35e720a2" containerName="istio-proxy"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154923    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc63ce5-6af2-4200-a4c0-86ffe970aab9" containerName="istio-proxy"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154927    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154933    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b767ec-a136-4b44-b740-79f223592863" containerName="payload-pre-processing"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154940    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="daee4916-972b-4f58-a7b5-9027dd759157" containerName="payload-processing"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154946    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154951    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154957    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154961    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154966    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154972    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="854f9bb0-0165-4bcd-8e45-fb91cf705f34" containerName="istio-proxy"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154976    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.154971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154982    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:54.155869 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154990    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:54.155869 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.154998    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:54.162564 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.162526    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.237784 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.233901    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dd840a-openshift-default-dockercfg-sn8v2\""
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380512    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380572    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380689    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380723    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380856    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380927    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.380979    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.381009    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcb8\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.382681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.381107    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.383981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.383954    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"]
Nov 01 21:00:54.493226 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493122    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493228    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493282    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493316    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcb8\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493533 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493391    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493533 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493480    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493533 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493519    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493685 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493604    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.493685 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.493640    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.494223 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.494068    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.494347 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.494250    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.494410 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.494366    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.494463 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.494409    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.494607 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.494579    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.495985 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.495963    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.496823 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.496796    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.551706 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.551620    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.552086 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.552058    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcb8\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8\") pod \"e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") " pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:54.805253 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:54.805170    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:55.285311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.285283    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"]
Nov 01 21:00:55.325171 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:00:55.325137    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf20bb71_b131_43ca_bd74_0e3ad32a05ed.slice/crio-3992edc573f0c927c330f4338977c26bc60d56db5ab3a905c186a92d17a46e22 WatchSource:0}: Error finding container 3992edc573f0c927c330f4338977c26bc60d56db5ab3a905c186a92d17a46e22: Status 404 returned error can't find the container with id 3992edc573f0c927c330f4338977c26bc60d56db5ab3a905c186a92d17a46e22
Nov 01 21:00:55.327813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.327782    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:00:55.327880 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.327852    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:00:55.327926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.327882    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.860165    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"]
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.870859    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.870881    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.870919    4872 cpu_manager.go:414] "containerMap: removing container" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.870926    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e35d60-b06f-4819-b15f-b9ade46dc5b9" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.870950    4872 cpu_manager.go:414] "containerMap: removing container" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.870958    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3989b761-01dc-4efc-9570-679548d75d21" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.871161    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f161503e-529a-420e-b4d7-f20372c59a7a" containerName="cleanup"
Nov 01 21:00:55.872683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.871176    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a483e8-0178-4c2b-ae88-ab55b5f9ec61" containerName="cleanup"
Nov 01 21:00:55.887678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.884035    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:55.907673 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.904884    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"]
Nov 01 21:00:55.951596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.951078    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d08fd0-dockercfg-s6r86\""
Nov 01 21:00:55.953074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.953044    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlkn\" (UniqueName: \"kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn\") pod \"payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:55.953219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.953132    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:55.971685 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:55.957837    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d08fd0\""
Nov 01 21:00:56.054489 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.054455    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:56.054835 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.054818    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlkn\" (UniqueName: \"kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn\") pod \"payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:56.055197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.055179    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:56.112750 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.112718    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlkn\" (UniqueName: \"kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn\") pod \"payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:56.232307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.232272    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" event={"ID":"cf20bb71-b131-43ca-bd74-0e3ad32a05ed","Type":"ContainerStarted","Data":"3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"}
Nov 01 21:00:56.232485 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.232334    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" event={"ID":"cf20bb71-b131-43ca-bd74-0e3ad32a05ed","Type":"ContainerStarted","Data":"3992edc573f0c927c330f4338977c26bc60d56db5ab3a905c186a92d17a46e22"}
Nov 01 21:00:56.265441 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.265405    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:56.358806 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.358752    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podStartSLOduration=2.35873273 podStartE2EDuration="2.35873273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:56.355118758 +0000 UTC m=+1813.404228576" watchObservedRunningTime="2026-11-01 21:00:56.35873273 +0000 UTC m=+1813.407842549"
Nov 01 21:00:56.649092 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.649061    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"]
Nov 01 21:00:56.649803 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:00:56.649777    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9688de2_bf9f_429e_a098_660232194e5b.slice/crio-9e583cb89d0a51e14811447827934301133727852d5dc96d349bfbad9be88cf8 WatchSource:0}: Error finding container 9e583cb89d0a51e14811447827934301133727852d5dc96d349bfbad9be88cf8: Status 404 returned error can't find the container with id 9e583cb89d0a51e14811447827934301133727852d5dc96d349bfbad9be88cf8
Nov 01 21:00:56.805743 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.805712    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:56.812028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:56.811995    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:57.242524 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:57.242493    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5" event={"ID":"c9688de2-bf9f-429e-a098-660232194e5b","Type":"ContainerStarted","Data":"d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00"}
Nov 01 21:00:57.242948 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:57.242545    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5" event={"ID":"c9688de2-bf9f-429e-a098-660232194e5b","Type":"ContainerStarted","Data":"9e583cb89d0a51e14811447827934301133727852d5dc96d349bfbad9be88cf8"}
Nov 01 21:00:57.242948 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:57.242788    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:57.242948 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:57.242825    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:00:57.243703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:57.243680    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:00:57.360812 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:00:57.360631    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5" podStartSLOduration=2.360611858 podStartE2EDuration="2.360611858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:57.358003377 +0000 UTC m=+1814.407113195" watchObservedRunningTime="2026-11-01 21:00:57.360611858 +0000 UTC m=+1814.409721677"
Nov 01 21:01:08.254985 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:08.254954    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:01:23.367428 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:01:23.367386    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:01:40.187552 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.187514    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"]
Nov 01 21:01:40.198685 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.194090    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.208679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.206637    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.208679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.207008    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlqj\" (UniqueName: \"kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj\") pod \"payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.287511 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.287464    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"]
Nov 01 21:01:40.292143 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.292100    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-081c8e\""
Nov 01 21:01:40.292501 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.292482    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-081c8e-dockercfg-bdsxx\""
Nov 01 21:01:40.308705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.308630    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlqj\" (UniqueName: \"kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj\") pod \"payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.309558 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.309530    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.312192 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.312170    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.347871 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.347835    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"]
Nov 01 21:01:40.353953 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.353763    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.370177 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.370084    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlqj\" (UniqueName: \"kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj\") pod \"payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.392986 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.392950    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"]
Nov 01 21:01:40.414406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.414220    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr466\" (UniqueName: \"kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466\") pod \"payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") " pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.418535 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.414835    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") " pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.516395 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.516272    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr466\" (UniqueName: \"kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466\") pod \"payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") " pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.516571 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.516533    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") " pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.517581 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.517553    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") " pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.574707 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.574633    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr466\" (UniqueName: \"kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466\") pod \"payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") " pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.612835 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.612788    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:40.676017 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.675990    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:40.996350 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:40.996321    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"]
Nov 01 21:01:40.997399 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:01:40.997374    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4867fd_e94a_4159_b66d_7c900001273a.slice/crio-2dffe43d33e1b3381577f647dd347fc6e95e76a492343e26c7030e15908eab7e WatchSource:0}: Error finding container 2dffe43d33e1b3381577f647dd347fc6e95e76a492343e26c7030e15908eab7e: Status 404 returned error can't find the container with id 2dffe43d33e1b3381577f647dd347fc6e95e76a492343e26c7030e15908eab7e
Nov 01 21:01:41.100686 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:01:41.099902    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b8b93e_f902_4396_bfd4_ddd85f6c7391.slice/crio-faa9f0269e2f4e9326b637409236a3729f3e726cfa0bbfe4736aef3f3261a590 WatchSource:0}: Error finding container faa9f0269e2f4e9326b637409236a3729f3e726cfa0bbfe4736aef3f3261a590: Status 404 returned error can't find the container with id faa9f0269e2f4e9326b637409236a3729f3e726cfa0bbfe4736aef3f3261a590
Nov 01 21:01:41.100686 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.100572    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"]
Nov 01 21:01:41.558000 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.557968    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm" event={"ID":"49b8b93e-f902-4396-bfd4-ddd85f6c7391","Type":"ContainerStarted","Data":"4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9"}
Nov 01 21:01:41.558474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.558024    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm" event={"ID":"49b8b93e-f902-4396-bfd4-ddd85f6c7391","Type":"ContainerStarted","Data":"faa9f0269e2f4e9326b637409236a3729f3e726cfa0bbfe4736aef3f3261a590"}
Nov 01 21:01:41.558474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.558176    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:41.559693 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.559637    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx" event={"ID":"6a4867fd-e94a-4159-b66d-7c900001273a","Type":"ContainerStarted","Data":"c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e"}
Nov 01 21:01:41.559808 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.559696    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx" event={"ID":"6a4867fd-e94a-4159-b66d-7c900001273a","Type":"ContainerStarted","Data":"2dffe43d33e1b3381577f647dd347fc6e95e76a492343e26c7030e15908eab7e"}
Nov 01 21:01:41.559808 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.559726    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:41.677758 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.677682    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm" podStartSLOduration=1.67764918 podStartE2EDuration="1.67764918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:41.675754782 +0000 UTC m=+1858.724864603" watchObservedRunningTime="2026-11-01 21:01:41.67764918 +0000 UTC m=+1858.726759048"
Nov 01 21:01:41.796290 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:41.796214    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx" podStartSLOduration=1.7961929749999999 podStartE2EDuration="1.796192975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:41.795076834 +0000 UTC m=+1858.844186655" watchObservedRunningTime="2026-11-01 21:01:41.796192975 +0000 UTC m=+1858.845302796"
Nov 01 21:01:48.880726 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:48.880692    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"]
Nov 01 21:01:48.886368 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:48.886345    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:48.990944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:48.990906    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-dab736-serving-cert\""
Nov 01 21:01:48.997278 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:48.997241    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"]
Nov 01 21:01:49.006266 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.006234    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pjn\" (UniqueName: \"kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.006412 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.006350    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.006556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.006535    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.006605 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.006593    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.108456 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.108419    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.108703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.108484    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.108703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.108566    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pjn\" (UniqueName: \"kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.108703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.108617    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.108887 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:01:49.108762    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-serving-cert: secret "maas-api-e2e-shared-b-w4-dab736-serving-cert" not found
Nov 01 21:01:49.108887 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:01:49.108831    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls podName:c2858fc5-b3f7-43e9-bfb0-392cabc19a3a nodeName:}" failed. No retries permitted until 2026-11-01 21:01:49.608813505 +0000 UTC m=+1866.657923304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls") pod "maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" (UID: "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a") : secret "maas-api-e2e-shared-b-w4-dab736-serving-cert" not found
Nov 01 21:01:49.109298 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.109276    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.111436 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.111412    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.167076 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.167043    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pjn\" (UniqueName: \"kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.621118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.621030    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.623666 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.623632    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:49.800762 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:49.800723    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:50.188554 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:50.188519    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"]
Nov 01 21:01:50.190436 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:01:50.190401    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2858fc5_b3f7_43e9_bfb0_392cabc19a3a.slice/crio-c05056118ffae06437458ea1f68d482625b36b1541d47a68a64fc4f0ca31e992 WatchSource:0}: Error finding container c05056118ffae06437458ea1f68d482625b36b1541d47a68a64fc4f0ca31e992: Status 404 returned error can't find the container with id c05056118ffae06437458ea1f68d482625b36b1541d47a68a64fc4f0ca31e992
Nov 01 21:01:50.192264 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:50.192241    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:01:50.628146 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:50.628050    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" event={"ID":"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a","Type":"ContainerStarted","Data":"c05056118ffae06437458ea1f68d482625b36b1541d47a68a64fc4f0ca31e992"}
Nov 01 21:01:51.637417 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:51.637380    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" event={"ID":"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a","Type":"ContainerStarted","Data":"cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2"}
Nov 01 21:01:51.638110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:51.637532    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:01:51.756242 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:51.756181    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" podStartSLOduration=2.953735381 podStartE2EDuration="3.756162768s" totalImagesPullingTime="802.427387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:51.753512634 +0000 UTC m=+1868.802622456" watchObservedRunningTime="2026-11-01 21:01:51.756162768 +0000 UTC m=+1868.805272585"
Nov 01 21:01:52.570567 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:52.570535    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:01:52.570943 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:52.570924    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:01:53.377920 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:01:53.377879    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:01:57.649832 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:01:57.649803    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:02:15.762212 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.762176    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"]
Nov 01 21:02:15.768110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.768086    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.802178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.802135    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9tq\" (UniqueName: \"kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.802625 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.802596    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.825247 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.825208    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"]
Nov 01 21:02:15.866169 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.866131    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-11a069b1-dockercfg-588d8\""
Nov 01 21:02:15.866362 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.866195    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-11a069b1\""
Nov 01 21:02:15.922678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.917930    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.922678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.918170    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9tq\" (UniqueName: \"kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.922678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.919457    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.978709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.974021    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"]
Nov 01 21:02:15.978709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.977485    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9tq\" (UniqueName: \"kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq\") pod \"payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:15.997574 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.997537    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"]
Nov 01 21:02:15.997795 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:15.997649    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.093187 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.092350    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:16.125428 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.122228    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxt55\" (UniqueName: \"kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.125428 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.122267    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.269776 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.269402    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxt55\" (UniqueName: \"kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.269776 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.269455    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.284091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.283926    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.327697 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.327621    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxt55\" (UniqueName: \"kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55\") pod \"payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") " pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.334273 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.334234    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.537556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.537523    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"]
Nov 01 21:02:16.537698 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:02:16.537626    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936bd6c6_10e6_4d18_8739_64d67c6610c3.slice/crio-2746b72fd753f8fba60eb6212ef44c39e8e007e962487a95ed92952865144449 WatchSource:0}: Error finding container 2746b72fd753f8fba60eb6212ef44c39e8e007e962487a95ed92952865144449: Status 404 returned error can't find the container with id 2746b72fd753f8fba60eb6212ef44c39e8e007e962487a95ed92952865144449
Nov 01 21:02:16.719422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.719393    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"]
Nov 01 21:02:16.720012 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:02:16.719986    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0198310b_397b_4a13_b99a_6783d85f140d.slice/crio-5a1b85d31150a6aac962b244268041d5d2a1b3f22dd6ecb955d276aa55c425bc WatchSource:0}: Error finding container 5a1b85d31150a6aac962b244268041d5d2a1b3f22dd6ecb955d276aa55c425bc: Status 404 returned error can't find the container with id 5a1b85d31150a6aac962b244268041d5d2a1b3f22dd6ecb955d276aa55c425bc
Nov 01 21:02:16.840556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.840515    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd" event={"ID":"0198310b-397b-4a13-b99a-6783d85f140d","Type":"ContainerStarted","Data":"c7a351427ea1aacf04f13d3983adcaaeb1ec6a7983cdbeaf30269f616675b911"}
Nov 01 21:02:16.840999 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.840611    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd" event={"ID":"0198310b-397b-4a13-b99a-6783d85f140d","Type":"ContainerStarted","Data":"5a1b85d31150a6aac962b244268041d5d2a1b3f22dd6ecb955d276aa55c425bc"}
Nov 01 21:02:16.840999 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.840798    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:16.842866 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.842839    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7" event={"ID":"936bd6c6-10e6-4d18-8739-64d67c6610c3","Type":"ContainerStarted","Data":"b989a50a6fe7ee31fee6a8268b410d9e668abb7110c4d4892a3a5ddcfb1eab94"}
Nov 01 21:02:16.842989 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.842893    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7" event={"ID":"936bd6c6-10e6-4d18-8739-64d67c6610c3","Type":"ContainerStarted","Data":"2746b72fd753f8fba60eb6212ef44c39e8e007e962487a95ed92952865144449"}
Nov 01 21:02:16.842989 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.842917    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:16.959419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:16.959368    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd" podStartSLOduration=1.95935566 podStartE2EDuration="1.95935566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:16.955694054 +0000 UTC m=+1894.004803887" watchObservedRunningTime="2026-11-01 21:02:16.95935566 +0000 UTC m=+1894.008465478"
Nov 01 21:02:17.080324 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:17.073545    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7" podStartSLOduration=2.073524185 podStartE2EDuration="2.073524185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:17.071073526 +0000 UTC m=+1894.120183384" watchObservedRunningTime="2026-11-01 21:02:17.073524185 +0000 UTC m=+1894.122634005"
Nov 01 21:02:23.375804 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:02:23.375759    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:02:27.852866 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:27.852834    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:02:27.853274 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:27.852968    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:02:45.093985 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:45.093950    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"]
Nov 01 21:02:45.094410 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:45.094171    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" containerID="cri-o://aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37" gracePeriod=30
Nov 01 21:02:45.429207 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:45.429175    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"]
Nov 01 21:02:46.081250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:46.081201    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" containerID="cri-o://7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f" gracePeriod=30
Nov 01 21:02:46.189485 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:46.189449    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"]
Nov 01 21:02:46.189916 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:46.189632    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" containerID="cri-o://9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9" gracePeriod=30
Nov 01 21:02:46.421956 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:46.421920    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:02:46.422129 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:46.421981    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:02:47.446845 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:47.446813    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:02:47.447298 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:47.446863    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:02:50.190637 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.190599    4872 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:02:50.191018 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.190749    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:02:50.416185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.416161    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 21:02:50.524214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524177    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524274    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524294    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524482 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524412    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524482 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524441    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524482 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524458    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:50.524640 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524523    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524640 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524595    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524640 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524608    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:50.524844 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524743    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524844 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524779    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:50.524967 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524845    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert\") pod \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\" (UID: \"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8\") "
Nov 01 21:02:50.524967 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.524919    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:50.525380 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.525354    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:50.525632 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.525609    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.525863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.525638    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.525863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.525669    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.525863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.525682    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.525863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.525695    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.526700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.526674    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:02:50.526891 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.526869    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:50.526977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.526899    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:50.527016 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.527000    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6" pod "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" (UID: "c2c247f1-be8b-42e2-8db0-2a0e3ba95df8"). InnerVolumeSpecName "kube-api-access-5mcf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:50.627579 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.627544    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcf6\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-kube-api-access-5mcf6\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.627579 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.627574    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.627579 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.627584    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:50.627834 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:50.627593    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.130695 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.130646    4872 generic.go:356] "Generic (PLEG): container finished" podID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerID="aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37" exitCode=0
Nov 01 21:02:51.130818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.130687    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" event={"ID":"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8","Type":"ContainerDied","Data":"aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37"}
Nov 01 21:02:51.130818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.130717    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"
Nov 01 21:02:51.130818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.130750    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n" event={"ID":"c2c247f1-be8b-42e2-8db0-2a0e3ba95df8","Type":"ContainerDied","Data":"3a067d293370de545b91404f0f6041a8da68d8f0af312272b0ccee83ab3f5ee0"}
Nov 01 21:02:51.130818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.130779    4872 scope.go:122] "RemoveContainer" containerID="aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37"
Nov 01 21:02:51.195647 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.195624    4872 scope.go:122] "RemoveContainer" containerID="aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37"
Nov 01 21:02:51.196258 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:02:51.196230    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37\": container with ID starting with aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37 not found: ID does not exist" containerID="aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37"
Nov 01 21:02:51.196361 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.196319    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37"} err="failed to get container status \"aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37\": rpc error: code = NotFound desc = could not find container \"aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37\": container with ID starting with aa3fff77e24dbf572044c8d4d570322e9ee9ded86cb50faf1b442569f2a9cb37 not found: ID does not exist"
Nov 01 21:02:51.353440 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.353299    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"]
Nov 01 21:02:51.358898 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.358851    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-wh67n"]
Nov 01 21:02:51.404963 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.404934    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 21:02:51.442226 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442188    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442295    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442371    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442444    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442503    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442536    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442723 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442594    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442784 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442733    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.442884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.442863    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs\") pod \"2174bfb3-d995-43a9-9616-495310f38b10\" (UID: \"2174bfb3-d995-43a9-9616-495310f38b10\") "
Nov 01 21:02:51.443286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443256    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.443286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443272    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.443448 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443322    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.443508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443458    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:51.443718 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443694    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.443802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443724    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.443802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443739    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2174bfb3-d995-43a9-9616-495310f38b10-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.443802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443753    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.444031 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.443873    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.445564 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.445525    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "kube-api-access-2rvzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:51.446947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.446576    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:51.446947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.446643    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.446947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.446899    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo" pod "2174bfb3-d995-43a9-9616-495310f38b10" (UID: "2174bfb3-d995-43a9-9616-495310f38b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:02:51.479335 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.479308    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 21:02:51.544913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.544866    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545115 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.544965    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545115 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.544995    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545115 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545017    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545119    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545236    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.545310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545255    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545292    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545335    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545348    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.545518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545368    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token\") pod \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\" (UID: \"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff\") "
Nov 01 21:02:51.545720 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545546    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.545792 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.545728    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546314    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546341    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546357    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2174bfb3-d995-43a9-9616-495310f38b10-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546371    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546385    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvzh\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-kube-api-access-2rvzh\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546402    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2174bfb3-d995-43a9-9616-495310f38b10-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546417    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546431    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546443    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2174bfb3-d995-43a9-9616-495310f38b10-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.546780 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.546745    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:51.547978 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.547950    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:02:51.548085 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.548003    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "kube-api-access-4mpss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:51.548243 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.548220    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:02:51.548307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.548229    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token" pod "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" (UID: "3b4b9217-3e7b-4f74-83fc-e269ff1a5bff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:51.647619 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.647586    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.647619 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.647618    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpss\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-kube-api-access-4mpss\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.647862 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.647628    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.647862 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.647639    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:51.647862 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:51.647649    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:52.139021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.138933    4872 generic.go:356] "Generic (PLEG): container finished" podID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerID="9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9" exitCode=0
Nov 01 21:02:52.139021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.139000    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"
Nov 01 21:02:52.139237 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.139018    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" event={"ID":"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff","Type":"ContainerDied","Data":"9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9"}
Nov 01 21:02:52.139237 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.139077    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc" event={"ID":"3b4b9217-3e7b-4f74-83fc-e269ff1a5bff","Type":"ContainerDied","Data":"8e9b450bf098a6af2416d6771ac02de2399ef59da2cb212769bebfb4bed0ba7a"}
Nov 01 21:02:52.139237 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.139108    4872 scope.go:122] "RemoveContainer" containerID="9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9"
Nov 01 21:02:52.140574 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.140549    4872 generic.go:356] "Generic (PLEG): container finished" podID="2174bfb3-d995-43a9-9616-495310f38b10" containerID="7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f" exitCode=0
Nov 01 21:02:52.140692 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.140619    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"
Nov 01 21:02:52.140804 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.140779    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" event={"ID":"2174bfb3-d995-43a9-9616-495310f38b10","Type":"ContainerDied","Data":"7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f"}
Nov 01 21:02:52.140864 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.140837    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k" event={"ID":"2174bfb3-d995-43a9-9616-495310f38b10","Type":"ContainerDied","Data":"69e4893d1aa60d68d7259b8decc034e5e86ee0e5d6017d30575ea563785647bf"}
Nov 01 21:02:52.150361 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.150342    4872 scope.go:122] "RemoveContainer" containerID="9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9"
Nov 01 21:02:52.150738 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:02:52.150710    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9\": container with ID starting with 9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9 not found: ID does not exist" containerID="9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9"
Nov 01 21:02:52.150837 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.150748    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9"} err="failed to get container status \"9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9\": rpc error: code = NotFound desc = could not find container \"9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9\": container with ID starting with 9e4484765db793549c64182146eb82e4a7b1713b542f2e82cb76629d76a9dcf9 not found: ID does not exist"
Nov 01 21:02:52.150837 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.150771    4872 scope.go:122] "RemoveContainer" containerID="7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f"
Nov 01 21:02:52.160710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.160690    4872 scope.go:122] "RemoveContainer" containerID="7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f"
Nov 01 21:02:52.160991 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:02:52.160965    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f\": container with ID starting with 7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f not found: ID does not exist" containerID="7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f"
Nov 01 21:02:52.161078 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.160994    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f"} err="failed to get container status \"7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f\": rpc error: code = NotFound desc = could not find container \"7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f\": container with ID starting with 7ab55c78826dfefd037892dceb604c7e3426f017c65f92b25a5af576ad97116f not found: ID does not exist"
Nov 01 21:02:52.307369 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.307332    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"]
Nov 01 21:02:52.311291 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.311263    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-q5fbc"]
Nov 01 21:02:52.425427 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.425394    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"]
Nov 01 21:02:52.428977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:52.428949    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-2bb5k"]
Nov 01 21:02:53.080673 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:53.080630    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2174bfb3-d995-43a9-9616-495310f38b10" path="/var/lib/kubelet/pods/2174bfb3-d995-43a9-9616-495310f38b10/volumes"
Nov 01 21:02:53.081314 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:53.081290    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" path="/var/lib/kubelet/pods/3b4b9217-3e7b-4f74-83fc-e269ff1a5bff/volumes"
Nov 01 21:02:53.081880 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:02:53.081860    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" path="/var/lib/kubelet/pods/c2c247f1-be8b-42e2-8db0-2a0e3ba95df8/volumes"
Nov 01 21:02:53.368559 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:02:53.368477    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:03:01.640437 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:01.640358    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"]
Nov 01 21:03:01.640805 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:01.640570    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" containerID="cri-o://ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26" gracePeriod=30
Nov 01 21:03:04.485815 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:04.485780    4872 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:03:04.486206 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:04.485834    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:03:06.923541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.923474    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 21:03:06.983824 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.983787    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.983982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.983834    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.983982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.983892    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.983982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.983912    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.983982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.983933    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.984219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.983988    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.984219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984050    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.984219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984194    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.984219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984195    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:06.984430 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984212    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:06.984430 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984271    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:06.984430 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984317    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert\") pod \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\" (UID: \"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6\") "
Nov 01 21:03:06.984593 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984553    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:06.984650 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.984635    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:06.985087 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.985064    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:06.985190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.985092    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:06.985190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.985106    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:06.985190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.985118    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:06.985190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.985132    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:06.986234 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.986201    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:06.986343 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.986281    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:06.986624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.986602    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:06.986698 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:06.986643    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc" pod "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" (UID: "48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6"). InnerVolumeSpecName "kube-api-access-jhbcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:07.087392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.087352    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:07.087392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.087394    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:07.087627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.087410    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbcc\" (UniqueName: \"kubernetes.io/projected/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-kube-api-access-jhbcc\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:07.087627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.087425    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:07.262481 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.262391    4872 generic.go:356] "Generic (PLEG): container finished" podID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerID="ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26" exitCode=0
Nov 01 21:03:07.262481 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.262456    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"
Nov 01 21:03:07.262733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.262480    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" event={"ID":"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6","Type":"ContainerDied","Data":"ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26"}
Nov 01 21:03:07.262733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.262531    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5" event={"ID":"48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6","Type":"ContainerDied","Data":"550e1034563e5ee7dc84675fc44dd30fc7a3b2b81c86393562fc00e980e5a1c8"}
Nov 01 21:03:07.262733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.262557    4872 scope.go:122] "RemoveContainer" containerID="ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26"
Nov 01 21:03:07.275924 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.275903    4872 scope.go:122] "RemoveContainer" containerID="ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26"
Nov 01 21:03:07.276269 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:07.276248    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26\": container with ID starting with ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26 not found: ID does not exist" containerID="ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26"
Nov 01 21:03:07.276320 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.276279    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26"} err="failed to get container status \"ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26\": rpc error: code = NotFound desc = could not find container \"ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26\": container with ID starting with ea766dc539464419127bf3396443282a2a5a627d96fa15d9602f449fce54ec26 not found: ID does not exist"
Nov 01 21:03:07.430959 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.430923    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"]
Nov 01 21:03:07.434936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:07.434912    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-8vhd5"]
Nov 01 21:03:09.081021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:09.080989    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" path="/var/lib/kubelet/pods/48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6/volumes"
Nov 01 21:03:18.904945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.904435    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"]
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906102    4872 cpu_manager.go:414] "containerMap: removing container" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906123    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906239    4872 cpu_manager.go:414] "containerMap: removing container" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906247    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906266    4872 cpu_manager.go:414] "containerMap: removing container" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906275    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906332    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906339    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906668    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="2174bfb3-d995-43a9-9616-495310f38b10" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906698    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bfa8d1-cccb-4d3c-b291-bd77f5d86ef6" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906728    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c247f1-be8b-42e2-8db0-2a0e3ba95df8" containerName="istio-proxy"
Nov 01 21:03:18.912351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.906747    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4b9217-3e7b-4f74-83fc-e269ff1a5bff" containerName="istio-proxy"
Nov 01 21:03:18.913963 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.913937    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:18.967352 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:18.967320    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-22ed7b-serving-cert\""
Nov 01 21:03:19.022063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.022030    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"]
Nov 01 21:03:19.074386 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.074351    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.074567 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.074398    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.074617 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.074594    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.074717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.074698    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mtg\" (UniqueName: \"kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.177492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.177211    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mtg\" (UniqueName: \"kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.177814 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.177557    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.177814 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.177586    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.180389 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.178555    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.180389 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.178773    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.180389 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:19.179642    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-serving-cert: secret "maas-api-e2e-api-a-22ed7b-serving-cert" not found
Nov 01 21:03:19.180389 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:19.179735    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls podName:4168885a-d9b3-473e-abd5-f283b05a5de0 nodeName:}" failed. No retries permitted until 2026-11-01 21:03:19.679715978 +0000 UTC m=+1956.728825775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls") pod "maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" (UID: "4168885a-d9b3-473e-abd5-f283b05a5de0") : secret "maas-api-e2e-api-a-22ed7b-serving-cert" not found
Nov 01 21:03:19.186079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.186057    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.236848 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.236816    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mtg\" (UniqueName: \"kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.686582 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.686543    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.689363 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.689335    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\") pod \"maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:19.829972 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:19.829939    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:20.217372 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:20.217336    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"]
Nov 01 21:03:20.217799 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:03:20.217688    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4168885a_d9b3_473e_abd5_f283b05a5de0.slice/crio-c681d0e9a18f74e171b16228320bcbf01b39559c5bd7d1981fd999a70ddbbdf5 WatchSource:0}: Error finding container c681d0e9a18f74e171b16228320bcbf01b39559c5bd7d1981fd999a70ddbbdf5: Status 404 returned error can't find the container with id c681d0e9a18f74e171b16228320bcbf01b39559c5bd7d1981fd999a70ddbbdf5
Nov 01 21:03:20.361569 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:20.361533    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" event={"ID":"4168885a-d9b3-473e-abd5-f283b05a5de0","Type":"ContainerStarted","Data":"c681d0e9a18f74e171b16228320bcbf01b39559c5bd7d1981fd999a70ddbbdf5"}
Nov 01 21:03:21.370433 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:21.370328    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" event={"ID":"4168885a-d9b3-473e-abd5-f283b05a5de0","Type":"ContainerStarted","Data":"bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800"}
Nov 01 21:03:21.370433 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:21.370409    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:21.488140 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:21.488085    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" podStartSLOduration=2.641841998 podStartE2EDuration="3.488069541s" totalImagesPullingTime="846.227543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:21.485551096 +0000 UTC m=+1958.534660917" watchObservedRunningTime="2026-11-01 21:03:21.488069541 +0000 UTC m=+1958.537179360"
Nov 01 21:03:23.369070 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:23.369035    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:03:27.383266 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:27.383238    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:03:30.061896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.061862    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"]
Nov 01 21:03:30.062308 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.062062    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" containerID="cri-o://7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb" gracePeriod=30
Nov 01 21:03:30.447267 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.447236    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"]
Nov 01 21:03:30.447463 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.447440    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" containerID="cri-o://a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326" gracePeriod=30
Nov 01 21:03:30.524975 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.524941    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:03:30.525125 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.525009    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:03:30.787668 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.787619    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"]
Nov 01 21:03:30.787872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.787831    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" containerID="cri-o://3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7" gracePeriod=30
Nov 01 21:03:30.863538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.863499    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"]
Nov 01 21:03:30.863820 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:30.863794    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy" containerID="cri-o://b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c" gracePeriod=30
Nov 01 21:03:31.120871 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.120782    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:03:31.121248 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.120866    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:03:31.147823 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.147777    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"]
Nov 01 21:03:31.148060 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.148030    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" containerID="cri-o://d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5" gracePeriod=30
Nov 01 21:03:31.152532 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.152501    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"]
Nov 01 21:03:31.152731 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.152713    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" containerID="cri-o://629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12" gracePeriod=30
Nov 01 21:03:31.280941 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.280902    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:03:31.281101 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:31.280968    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:03:32.893518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:32.893481    4872 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:03:32.893984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:32.893545    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:03:34.103255 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:34.103217    4872 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@7-26840-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf20bb71_b131_43ca_bd74_0e3ad32a05ed.slice/crio-3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 21:03:34.494241 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:34.494207    4872 generic.go:356] "Generic (PLEG): container finished" podID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerID="3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b" exitCode=0
Nov 01 21:03:34.494410 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:34.494279    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" event={"ID":"cf20bb71-b131-43ca-bd74-0e3ad32a05ed","Type":"ContainerDied","Data":"3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"}
Nov 01 21:03:34.494773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:34.494759    4872 scope.go:122] "RemoveContainer" containerID="3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"
Nov 01 21:03:34.496489 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:34.496454    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:03:34.496599 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:34.496542    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:03:34.496679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:34.496608    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:03:35.167682 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.166184    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:03:35.167682 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.166261    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:03:35.385061 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.385033    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 21:03:35.533190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.531932    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" event={"ID":"cf20bb71-b131-43ca-bd74-0e3ad32a05ed","Type":"ContainerStarted","Data":"997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892"}
Nov 01 21:03:35.535974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.535945    4872 generic.go:356] "Generic (PLEG): container finished" podID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerID="7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb" exitCode=0
Nov 01 21:03:35.536106 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.535997    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"
Nov 01 21:03:35.536106 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.536032    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" event={"ID":"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e","Type":"ContainerDied","Data":"7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb"}
Nov 01 21:03:35.536106 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.536079    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth" event={"ID":"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e","Type":"ContainerDied","Data":"eda350f5e4ede2f7ff5c437cb8bc6edfef12c87679ff4f5c8a89408c92789227"}
Nov 01 21:03:35.536106 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.536101    4872 scope.go:122] "RemoveContainer" containerID="7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb"
Nov 01 21:03:35.537545 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537243    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537545 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537345    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537545 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537421    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537545 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537467    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537545 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537539    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537857 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537615    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537857 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537673    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537857 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537679    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:35.537857 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537733    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.537857 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537824    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww\") pod \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\" (UID: \"a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e\") "
Nov 01 21:03:35.538095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537849    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.538095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.537998    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.538095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538022    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.538794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538171    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.538794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538518    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.538794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538539    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.538794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538554    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.538794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538568    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.538794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.538582    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.544568 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.544542    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.544678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.544545    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:35.546050 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.546031    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "kube-api-access-5xgww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:35.546126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.546059    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token" pod "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" (UID: "a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:35.565877 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.565769    4872 scope.go:122] "RemoveContainer" containerID="7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb"
Nov 01 21:03:35.566160 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:35.566126    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb\": container with ID starting with 7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb not found: ID does not exist" containerID="7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb"
Nov 01 21:03:35.566233 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.566161    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb"} err="failed to get container status \"7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb\": rpc error: code = NotFound desc = could not find container \"7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb\": container with ID starting with 7a775ac1e189e1f68401eccf74aaa8e820c6ce0dfed983c77beb51fbcda40fbb not found: ID does not exist"
Nov 01 21:03:35.640939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.640851    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.640939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.640883    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.640939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.640898    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.640939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.640913    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xgww\" (UniqueName: \"kubernetes.io/projected/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e-kube-api-access-5xgww\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.761482 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.761458    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 21:03:35.811678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.809040    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:03:35.811678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.809177    4872 patch_prober.go:29] interesting pod/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 21:03:35.811678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.809220    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 21:03:35.843877 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.843841    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844018    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844111 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844058    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844171 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844106    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844171 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844152    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844203    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844220    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:35.844398 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844286    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844398 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844349    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844505 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844460    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket\") pod \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\" (UID: \"f2c28b06-dce7-47b3-8b9b-2501d63f7463\") "
Nov 01 21:03:35.844505 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844481    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.844747 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844722    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.844747 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844738    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.844950 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.844928    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.845279 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.845254    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.845368 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.845286    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.845368 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.845303    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.845368 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.845318    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.845368 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.845332    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.847334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.847308    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:35.847434 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.847387    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:35.847493 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.847479    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:35.847911 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.847875    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq" pod "f2c28b06-dce7-47b3-8b9b-2501d63f7463" (UID: "f2c28b06-dce7-47b3-8b9b-2501d63f7463"). InnerVolumeSpecName "kube-api-access-6mbhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:35.946255 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.946222    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.946255 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.946252    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.946460 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.946266    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:35.946460 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:35.946278    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mbhq\" (UniqueName: \"kubernetes.io/projected/f2c28b06-dce7-47b3-8b9b-2501d63f7463-kube-api-access-6mbhq\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.028162 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.028128    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"]
Nov 01 21:03:36.034702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.034668    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gkcth"]
Nov 01 21:03:36.382861 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.382828    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 21:03:36.389956 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.389893    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.454383    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.454448    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.454499    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.454554    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.454588    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.454634    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.455174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.455009    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.455641 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.455189    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456013    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456207    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456239    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456312    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456338    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456360    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456401    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456425    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456467    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456496    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456540    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket\") pod \"c0c263b9-107a-4225-9fa9-21a960aa8c20\" (UID: \"c0c263b9-107a-4225-9fa9-21a960aa8c20\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456644    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.456707    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c\") pod \"968359e6-daa8-4326-958f-9aebb9837f10\" (UID: \"968359e6-daa8-4326-958f-9aebb9837f10\") "
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.457496    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.457519    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.457533    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.458403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.458351    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:36.459454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.458863    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.459454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.459064    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.459454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.459422    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.461353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.460434    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:36.461353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.460561    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.461353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.460798    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.461353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.460883    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.461353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.461084    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "kube-api-access-9bcqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.461353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.461225    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.462702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.462306    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.462702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.462538    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "kube-api-access-xq92c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.462702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.462594    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo" pod "c0c263b9-107a-4225-9fa9-21a960aa8c20" (UID: "c0c263b9-107a-4225-9fa9-21a960aa8c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:36.463030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.463009    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:36.463516 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.463487    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy" pod "968359e6-daa8-4326-958f-9aebb9837f10" (UID: "968359e6-daa8-4326-958f-9aebb9837f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.476737 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.474382    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 21:03:36.538737 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.538705    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 21:03:36.545914 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.545883    4872 generic.go:356] "Generic (PLEG): container finished" podID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerID="a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326" exitCode=0
Nov 01 21:03:36.546132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.545959    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"
Nov 01 21:03:36.546132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.545993    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" event={"ID":"f2c28b06-dce7-47b3-8b9b-2501d63f7463","Type":"ContainerDied","Data":"a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326"}
Nov 01 21:03:36.546132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.546032    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42" event={"ID":"f2c28b06-dce7-47b3-8b9b-2501d63f7463","Type":"ContainerDied","Data":"bcc90010e0656ca879622fe4663f9fff7f07cb1eea1fcee988893a920bc8e0b4"}
Nov 01 21:03:36.546132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.546055    4872 scope.go:122] "RemoveContainer" containerID="a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326"
Nov 01 21:03:36.547765 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.547747    4872 generic.go:356] "Generic (PLEG): container finished" podID="968359e6-daa8-4326-958f-9aebb9837f10" containerID="3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7" exitCode=0
Nov 01 21:03:36.547896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.547808    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" event={"ID":"968359e6-daa8-4326-958f-9aebb9837f10","Type":"ContainerDied","Data":"3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7"}
Nov 01 21:03:36.547896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.547836    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97" event={"ID":"968359e6-daa8-4326-958f-9aebb9837f10","Type":"ContainerDied","Data":"8d64319873c87da880d94975efcd2ea4a7fd0d5dd4762df764a3764658922b11"}
Nov 01 21:03:36.547896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.547891    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"
Nov 01 21:03:36.553371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.553341    4872 generic.go:356] "Generic (PLEG): container finished" podID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerID="629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12" exitCode=0
Nov 01 21:03:36.553504 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.553442    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" event={"ID":"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa","Type":"ContainerDied","Data":"629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12"}
Nov 01 21:03:36.553504 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.553482    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n" event={"ID":"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa","Type":"ContainerDied","Data":"0e88c324d864bec7996739d7fa8ae8710bcca17acf94013eb220e619e6f56792"}
Nov 01 21:03:36.553627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.553545    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"
Nov 01 21:03:36.554844 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.554823    4872 generic.go:356] "Generic (PLEG): container finished" podID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerID="d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5" exitCode=0
Nov 01 21:03:36.555021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.554997    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" event={"ID":"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d","Type":"ContainerDied","Data":"d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5"}
Nov 01 21:03:36.555114 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.555046    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2" event={"ID":"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d","Type":"ContainerDied","Data":"d32bae0c1240c7fdc50cc1f60e130f0245e704037feb24e10d55d1dc8ad408ea"}
Nov 01 21:03:36.555204 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.555189    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"
Nov 01 21:03:36.558107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.557336    4872 generic.go:356] "Generic (PLEG): container finished" podID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerID="b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c" exitCode=0
Nov 01 21:03:36.558107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.557353    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" event={"ID":"c0c263b9-107a-4225-9fa9-21a960aa8c20","Type":"ContainerDied","Data":"b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c"}
Nov 01 21:03:36.558107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.558059    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd" event={"ID":"c0c263b9-107a-4225-9fa9-21a960aa8c20","Type":"ContainerDied","Data":"f9417648f553bdc841ea610dba88347cdf2d186932096f4bd3c6518f6d0c508f"}
Nov 01 21:03:36.558107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.557381    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"
Nov 01 21:03:36.558757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.558737    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.559297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.559188    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:36.559456 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.559439    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.559961 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.559844    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.560161 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.560145    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.560929 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.560913    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.561510 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.561262    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.561712 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.561696    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.562585 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.562290    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.562775 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.562492    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.562775 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.562565    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.563064 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.563045    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.563284 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.563270    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.563777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.563562    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.563777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.563670    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.564109 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.564096    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo\") pod \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\" (UID: \"3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d\") "
Nov 01 21:03:36.565764 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565735    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565887    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565912    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565929    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565945    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565961    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565975    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq92c\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-kube-api-access-xq92c\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.565991    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566005    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566020    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968359e6-daa8-4326-958f-9aebb9837f10-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566034    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566049    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566065    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566078    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c263b9-107a-4225-9fa9-21a960aa8c20-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566091    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566105    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcqg\" (UniqueName: \"kubernetes.io/projected/c0c263b9-107a-4225-9fa9-21a960aa8c20-kube-api-access-9bcqg\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566124    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968359e6-daa8-4326-958f-9aebb9837f10-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566139    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968359e6-daa8-4326-958f-9aebb9837f10-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566152    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c263b9-107a-4225-9fa9-21a960aa8c20-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566166    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566182    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566198    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968359e6-daa8-4326-958f-9aebb9837f10-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.566353 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.566212    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c263b9-107a-4225-9fa9-21a960aa8c20-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.571041 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.571001    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd" pod "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" (UID: "3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d"). InnerVolumeSpecName "kube-api-access-8tdwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.572135 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.572107    4872 scope.go:122] "RemoveContainer" containerID="a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326"
Nov 01 21:03:36.572587 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:36.572419    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326\": container with ID starting with a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326 not found: ID does not exist" containerID="a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326"
Nov 01 21:03:36.572587 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.572459    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326"} err="failed to get container status \"a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326\": rpc error: code = NotFound desc = could not find container \"a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326\": container with ID starting with a8f3a7dddeddbe201d459f446ee9ab61df49f6203a29b36273328b79783c7326 not found: ID does not exist"
Nov 01 21:03:36.572587 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.572485    4872 scope.go:122] "RemoveContainer" containerID="3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7"
Nov 01 21:03:36.583667 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.583638    4872 scope.go:122] "RemoveContainer" containerID="3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7"
Nov 01 21:03:36.583996 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:36.583978    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7\": container with ID starting with 3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7 not found: ID does not exist" containerID="3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7"
Nov 01 21:03:36.584079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.584002    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7"} err="failed to get container status \"3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7\": rpc error: code = NotFound desc = could not find container \"3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7\": container with ID starting with 3f5f1818292bf2ae14f0a818781577b6b322e70b803312b563e1d448201b00a7 not found: ID does not exist"
Nov 01 21:03:36.584079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.584020    4872 scope.go:122] "RemoveContainer" containerID="629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12"
Nov 01 21:03:36.593149 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.593132    4872 scope.go:122] "RemoveContainer" containerID="629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12"
Nov 01 21:03:36.593422 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:36.593404    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12\": container with ID starting with 629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12 not found: ID does not exist" containerID="629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12"
Nov 01 21:03:36.593489 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.593428    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12"} err="failed to get container status \"629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12\": rpc error: code = NotFound desc = could not find container \"629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12\": container with ID starting with 629b4fb486fcf166b8add2c8bcf23e3743a11d9b21902d78b992b6ed81c3ed12 not found: ID does not exist"
Nov 01 21:03:36.593489 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.593443    4872 scope.go:122] "RemoveContainer" containerID="d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5"
Nov 01 21:03:36.632702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.632679    4872 scope.go:122] "RemoveContainer" containerID="d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5"
Nov 01 21:03:36.633056 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:36.633032    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5\": container with ID starting with d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5 not found: ID does not exist" containerID="d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5"
Nov 01 21:03:36.633126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.633065    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5"} err="failed to get container status \"d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5\": rpc error: code = NotFound desc = could not find container \"d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5\": container with ID starting with d3ca71938ab0c9924646cdf0a4d376ebf5f6283275b6466f19e2ce663916e3c5 not found: ID does not exist"
Nov 01 21:03:36.633126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.633089    4872 scope.go:122] "RemoveContainer" containerID="b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c"
Nov 01 21:03:36.645624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.644558    4872 scope.go:122] "RemoveContainer" containerID="b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c"
Nov 01 21:03:36.652188 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:36.652150    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c\": container with ID starting with b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c not found: ID does not exist" containerID="b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c"
Nov 01 21:03:36.652407 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.652377    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c"} err="failed to get container status \"b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c\": rpc error: code = NotFound desc = could not find container \"b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c\": container with ID starting with b6eee75118ea62b4f602afb4c68f8bd1c161c054c6068df6147bc96a00fce40c not found: ID does not exist"
Nov 01 21:03:36.667722 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.667689    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.667933 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.667921    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668075 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668066    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668144 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668136    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668213 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668205    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668318 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668310    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668380 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668372    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668477 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668469    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.668574 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.668565    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92\") pod \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\" (UID: \"05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa\") "
Nov 01 21:03:36.669147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.669129    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdwd\" (UniqueName: \"kubernetes.io/projected/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-kube-api-access-8tdwd\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.669267 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.669256    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.670294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.670245    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.670441 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.670410    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.670507 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.670470    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.671026 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.670999    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:03:36.672044 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.672005    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.672517 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.672482    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:03:36.672686 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.672667    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:03:36.672805 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.672781    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "kube-api-access-lht92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.673898 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.673875    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token" pod "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" (UID: "05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.770997    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771027    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lht92\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-kube-api-access-lht92\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771038    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771049    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771063    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771071    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771080    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771413 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771090    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.771413 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.771103    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:03:36.820714 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.819582    4872 patch_prober.go:29] interesting pod/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 21:03:36.820714 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.819642    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 21:03:36.893187 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.893147    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"]
Nov 01 21:03:36.922454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:36.922410    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-z5s97"]
Nov 01 21:03:37.022926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.022822    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"]
Nov 01 21:03:37.025971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.025949    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-hdc42"]
Nov 01 21:03:37.087065 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.087030    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968359e6-daa8-4326-958f-9aebb9837f10" path="/var/lib/kubelet/pods/968359e6-daa8-4326-958f-9aebb9837f10/volumes"
Nov 01 21:03:37.088007 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.087984    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" path="/var/lib/kubelet/pods/a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e/volumes"
Nov 01 21:03:37.088825 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.088808    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" path="/var/lib/kubelet/pods/f2c28b06-dce7-47b3-8b9b-2501d63f7463/volumes"
Nov 01 21:03:37.134839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.134803    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"]
Nov 01 21:03:37.138556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.138535    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-49zgd"]
Nov 01 21:03:37.432908 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.432878    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"]
Nov 01 21:03:37.436046 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.436022    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-brks2"]
Nov 01 21:03:37.548836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.548803    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"]
Nov 01 21:03:37.551948 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.551923    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lrl8n"]
Nov 01 21:03:37.814839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.814748    4872 patch_prober.go:29] interesting pod/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 21:03:37.815085 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:37.815055    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 21:03:38.815333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:38.815299    4872 patch_prober.go:29] interesting pod/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 21:03:38.815960 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:38.815931    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 21:03:39.084739 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:39.084645    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" path="/var/lib/kubelet/pods/05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa/volumes"
Nov 01 21:03:39.085267 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:39.085249    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" path="/var/lib/kubelet/pods/3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d/volumes"
Nov 01 21:03:39.085757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:39.085741    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" path="/var/lib/kubelet/pods/c0c263b9-107a-4225-9fa9-21a960aa8c20/volumes"
Nov 01 21:03:39.816680 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:39.812060    4872 patch_prober.go:29] interesting pod/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 21:03:39.816680 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:39.812118    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 21:03:40.810271 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:40.810242    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:03:40.810540 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:40.810518    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:03:40.811190 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:40.811173    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:03:49.154121 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154080    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"]
Nov 01 21:03:49.154738 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154724    4872 cpu_manager.go:414] "containerMap: removing container" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154739    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154752    4872 cpu_manager.go:414] "containerMap: removing container" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154757    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154771    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154776    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154784    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy"
Nov 01 21:03:49.154797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154790    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154807    4872 cpu_manager.go:414] "containerMap: removing container" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154812    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154834    4872 cpu_manager.go:414] "containerMap: removing container" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154847    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154936    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbec3a4-9f96-41b4-94b5-2fdc0f8e803d" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154950    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="968359e6-daa8-4326-958f-9aebb9837f10" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154957    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cbb5f2-8cbf-44e2-b5c0-8f54cb6bb8fa" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154964    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b71b26-13c2-4d9a-885d-a7c3be6a9a3e" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154970    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c28b06-dce7-47b3-8b9b-2501d63f7463" containerName="istio-proxy"
Nov 01 21:03:49.155027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.154975    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c263b9-107a-4225-9fa9-21a960aa8c20" containerName="istio-proxy"
Nov 01 21:03:49.160068 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.160047    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.201431 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.201393    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.201618 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.201547    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.201676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.201613    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.201718 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.201693    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpcd\" (UniqueName: \"kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.213141 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.213105    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-11a069b1-serving-cert\""
Nov 01 21:03:49.216037 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.216003    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"]
Nov 01 21:03:49.303576 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.303523    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.303781 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.303593    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpcd\" (UniqueName: \"kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.303781 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:49.303739    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-serving-cert: secret "maas-api-e2e-mt-11a069b1-serving-cert" not found
Nov 01 21:03:49.303937 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:49.303839    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls podName:275ee54a-e089-40ed-96e3-4d2251b8c658 nodeName:}" failed. No retries permitted until 2026-11-01 21:03:49.803820848 +0000 UTC m=+1986.852930664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls") pod "maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" (UID: "275ee54a-e089-40ed-96e3-4d2251b8c658") : secret "maas-api-e2e-mt-11a069b1-serving-cert" not found
Nov 01 21:03:49.303937 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.303886    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.304053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.304021    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.304875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.304845    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.307011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.306985    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.363952 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.363924    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpcd\" (UniqueName: \"kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.814068 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.814036    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:49.816721 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:49.816697    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\") pod \"maas-api-e2e-mt-11a069b1-675bb68865-t7xcg\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:50.076014 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:50.074879    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:50.477903 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:50.477878    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"]
Nov 01 21:03:50.480857 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:03:50.480832    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275ee54a_e089_40ed_96e3_4d2251b8c658.slice/crio-08f57b59147139a22f75f18e0bb534d228dba1b1fe264a3549ecd24eb5894d63 WatchSource:0}: Error finding container 08f57b59147139a22f75f18e0bb534d228dba1b1fe264a3549ecd24eb5894d63: Status 404 returned error can't find the container with id 08f57b59147139a22f75f18e0bb534d228dba1b1fe264a3549ecd24eb5894d63
Nov 01 21:03:50.663861 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:50.663822    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" event={"ID":"275ee54a-e089-40ed-96e3-4d2251b8c658","Type":"ContainerStarted","Data":"08f57b59147139a22f75f18e0bb534d228dba1b1fe264a3549ecd24eb5894d63"}
Nov 01 21:03:51.672272 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:51.672235    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" event={"ID":"275ee54a-e089-40ed-96e3-4d2251b8c658","Type":"ContainerStarted","Data":"4f776ea95a15fb1bc0fcd6cbccae3c5aa111b34d0a59e94d9bc85f90ec9448b2"}
Nov 01 21:03:51.672695 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:51.672323    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:03:51.788051 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:51.787999    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" podStartSLOduration=1.915389768 podStartE2EDuration="2.787983681s" totalImagesPullingTime="872.593913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:51.787262631 +0000 UTC m=+1988.836372448" watchObservedRunningTime="2026-11-01 21:03:51.787983681 +0000 UTC m=+1988.837093499"
Nov 01 21:03:53.372696 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:03:53.372632    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:03:57.684194 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:03:57.684164    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:04:00.575394 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:00.575360    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"]
Nov 01 21:04:00.575928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:00.575615    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" containerID="cri-o://ab3f563bfeecac2cd0e929fe46c2b6d4e9b2752137045076b36bba1ce90b7649" gracePeriod=30
Nov 01 21:04:05.472029 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:05.471994    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-82z67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:04:05.472417 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:05.472060    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:04:05.775427 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:05.773962    4872 generic.go:356] "Generic (PLEG): container finished" podID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerID="ab3f563bfeecac2cd0e929fe46c2b6d4e9b2752137045076b36bba1ce90b7649" exitCode=0
Nov 01 21:04:05.775427 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:05.774149    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" event={"ID":"5f2444cc-8b9f-42de-8632-d903c7923a2c","Type":"ContainerDied","Data":"ab3f563bfeecac2cd0e929fe46c2b6d4e9b2752137045076b36bba1ce90b7649"}
Nov 01 21:04:05.863752 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:05.863721    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 21:04:06.037173 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037083    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.037332 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037242    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.037332 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037284    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.037332 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037317    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.037551 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037349    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.037551 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037358    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:04:06.037551 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037391    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.037551 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037466    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037562    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037566    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.037864    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs\") pod \"5f2444cc-8b9f-42de-8632-d903c7923a2c\" (UID: \"5f2444cc-8b9f-42de-8632-d903c7923a2c\") "
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.038002    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.038466    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.038578    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.038623    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2444cc-8b9f-42de-8632-d903c7923a2c-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.038638    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.038971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.038816    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:04:06.040249 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.040224    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "kube-api-access-6cr9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:04:06.040500 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.040479    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:04:06.040774 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.040750    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:04:06.040875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.040857    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy" pod "5f2444cc-8b9f-42de-8632-d903c7923a2c" (UID: "5f2444cc-8b9f-42de-8632-d903c7923a2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:04:06.141735 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.141702    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.141899 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.141757    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.141899 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.141774    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.141899 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.141791    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.141899 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.141806    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2444cc-8b9f-42de-8632-d903c7923a2c-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.141899 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.141820    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cr9d\" (UniqueName: \"kubernetes.io/projected/5f2444cc-8b9f-42de-8632-d903c7923a2c-kube-api-access-6cr9d\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:06.783492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.783462    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"
Nov 01 21:04:06.783972 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.783460    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67" event={"ID":"5f2444cc-8b9f-42de-8632-d903c7923a2c","Type":"ContainerDied","Data":"03d006cb4874ef61b1f266ceb8e01177db2d5a4c3bc15acb526b24c0a64822dd"}
Nov 01 21:04:06.783972 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:06.783621    4872 scope.go:122] "RemoveContainer" containerID="ab3f563bfeecac2cd0e929fe46c2b6d4e9b2752137045076b36bba1ce90b7649"
Nov 01 21:04:07.004381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:07.004340    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"]
Nov 01 21:04:07.054125 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:07.054049    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-82z67"]
Nov 01 21:04:07.090003 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:07.089799    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" path="/var/lib/kubelet/pods/5f2444cc-8b9f-42de-8632-d903c7923a2c/volumes"
Nov 01 21:04:23.367526 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:04:23.367489    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:04:34.343746 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.343642    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"]
Nov 01 21:04:34.344247 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.344232    4872 cpu_manager.go:414] "containerMap: removing container" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy"
Nov 01 21:04:34.344247 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.344246    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy"
Nov 01 21:04:34.344383 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.344373    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2444cc-8b9f-42de-8632-d903c7923a2c" containerName="istio-proxy"
Nov 01 21:04:34.349317 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.349293    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.401968 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.401940    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-11a069b1-openshift-default-dockercfg-c82c9\""
Nov 01 21:04:34.505236 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505201    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505236 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505238    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505453 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505303    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505453 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505399    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505520 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505504    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505558 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505537    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505605 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505564    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505691    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.505832 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.505814    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.548218 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.548175    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"]
Nov 01 21:04:34.607297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607215    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607248    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607287    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607310    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607367    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607398    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607431    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607619    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607812    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.607833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607819    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.608000 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.607943    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.608056 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.608002    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.608159 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.608137    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.608231 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.608174    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.609689 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.609645    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.610048 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.610028    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.665761 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.665730    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.666025 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.666000    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:34.963232 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:34.963196    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:35.399355 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:35.399323    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"]
Nov 01 21:04:35.400899 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:04:35.400872    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2685675b_088e_4fd8_80d0_373afccd70d0.slice/crio-4b060d8de828b9d73a758038d03b0ddb3a1f713ce8593b1737702587522142c2 WatchSource:0}: Error finding container 4b060d8de828b9d73a758038d03b0ddb3a1f713ce8593b1737702587522142c2: Status 404 returned error can't find the container with id 4b060d8de828b9d73a758038d03b0ddb3a1f713ce8593b1737702587522142c2
Nov 01 21:04:35.403565 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:35.403534    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:04:35.403646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:35.403599    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:04:35.403646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:35.403627    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:04:36.012622 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:36.012571    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" event={"ID":"2685675b-088e-4fd8-80d0-373afccd70d0","Type":"ContainerStarted","Data":"15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1"}
Nov 01 21:04:36.012823 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:36.012705    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" event={"ID":"2685675b-088e-4fd8-80d0-373afccd70d0","Type":"ContainerStarted","Data":"4b060d8de828b9d73a758038d03b0ddb3a1f713ce8593b1737702587522142c2"}
Nov 01 21:04:36.183432 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:36.183376    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podStartSLOduration=2.183361358 podStartE2EDuration="2.183361358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:36.182002107 +0000 UTC m=+2033.231111930" watchObservedRunningTime="2026-11-01 21:04:36.183361358 +0000 UTC m=+2033.232471240"
Nov 01 21:04:36.963627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:36.963596    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:36.965417 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:36.965388    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 21:04:36.965549 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:36.965449    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 21:04:37.963488 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:37.963454    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 21:04:37.963692 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:37.963518    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 21:04:38.964081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:38.964037    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 21:04:38.964488 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:38.964112    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 21:04:39.964605 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:39.964568    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 21:04:39.965110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:39.964702    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 21:04:40.963621 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:40.963573    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 21:04:40.963815 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:40.963650    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 21:04:41.963557 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:41.963524    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 01 21:04:41.964135 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:41.964100    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 01 21:04:42.968108 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:42.968080    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:42.968523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:42.968481    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:42.969274 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:42.969253    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:04:49.463407 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.463369    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"]
Nov 01 21:04:49.468926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.468903    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.571568 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.571523    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.571781 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.571688    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.571874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.571810    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.571874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.571863    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.571984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.571947    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.572038 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.572013    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.572116 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.572096    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.572177 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.572147    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg99n\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.572237 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.572220    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.578632 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.578604    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"]
Nov 01 21:04:49.673470 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673428    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673645 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673498    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673645 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673557    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673747 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673683    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg99n\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673798 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673779    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673864 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673848    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673911    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.673966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.673844    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.674026 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.674008    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.674087 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.674070    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.674156 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.674134    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.674400 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.674362    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.674494 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.674425    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.674555 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.674535    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.675958 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.675941    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.676207 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.676191    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.731176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.731106    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.731311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.731180    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg99n\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:49.782415 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:49.782384    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:50.213099 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:50.213067    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"]
Nov 01 21:04:50.213965 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:04:50.213943    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835b0743_70c1_4021_9655_ac6faadd38f9.slice/crio-0d2b438f4ffe8b1ccd7f5a4e0ed130b9813a3602e25de49f6350568a7a8ebaa6 WatchSource:0}: Error finding container 0d2b438f4ffe8b1ccd7f5a4e0ed130b9813a3602e25de49f6350568a7a8ebaa6: Status 404 returned error can't find the container with id 0d2b438f4ffe8b1ccd7f5a4e0ed130b9813a3602e25de49f6350568a7a8ebaa6
Nov 01 21:04:50.217415 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:50.217381    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:04:50.217503 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:50.217447    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:04:50.217503 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:50.217477    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:04:51.133180 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:51.133142    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" event={"ID":"835b0743-70c1-4021-9655-ac6faadd38f9","Type":"ContainerStarted","Data":"e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4"}
Nov 01 21:04:51.133598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:51.133208    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" event={"ID":"835b0743-70c1-4021-9655-ac6faadd38f9","Type":"ContainerStarted","Data":"0d2b438f4ffe8b1ccd7f5a4e0ed130b9813a3602e25de49f6350568a7a8ebaa6"}
Nov 01 21:04:51.335065 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:51.335003    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podStartSLOduration=2.334989397 podStartE2EDuration="2.334989397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:51.334825773 +0000 UTC m=+2048.383935593" watchObservedRunningTime="2026-11-01 21:04:51.334989397 +0000 UTC m=+2048.384099236"
Nov 01 21:04:51.783152 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:51.783119    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:51.784723 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:51.784694    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 21:04:51.784864 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:51.784743    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 21:04:52.722796 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:52.722755    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"]
Nov 01 21:04:52.783598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:52.783559    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 21:04:52.783786 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:52.783621    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 21:04:52.834344 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:52.834303    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"]
Nov 01 21:04:53.368723 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:04:53.368684    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:04:53.783388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:53.783355    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 21:04:53.783878 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:53.783415    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 21:04:54.782630 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:54.782597    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 21:04:54.782900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:54.782650    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 21:04:55.783539 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:55.783505    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 01 21:04:55.783964 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:55.783567    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 01 21:04:56.937609 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:56.937578    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:56.938104 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:56.938087    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:56.938856 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:56.938837    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:04:57.723982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:57.723939    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm" podUID="49b8b93e-f902-4396-bfd4-ddd85f6c7391" containerName="payload-processing" containerID="cri-o://4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9" gracePeriod=25
Nov 01 21:04:57.835194 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:57.835127    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx" podUID="6a4867fd-e94a-4159-b66d-7c900001273a" containerName="payload-pre-processing" containerID="cri-o://c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e" gracePeriod=25
Nov 01 21:04:58.086742 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.086520    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:04:58.094541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.094508    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume\") pod \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") "
Nov 01 21:04:58.094696 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.094576    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466\" (UniqueName: \"kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466\") pod \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\" (UID: \"49b8b93e-f902-4396-bfd4-ddd85f6c7391\") "
Nov 01 21:04:58.095911 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.095884    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume" pod "49b8b93e-f902-4396-bfd4-ddd85f6c7391" (UID: "49b8b93e-f902-4396-bfd4-ddd85f6c7391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:04:58.121088 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.121048    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466" pod "49b8b93e-f902-4396-bfd4-ddd85f6c7391" (UID: "49b8b93e-f902-4396-bfd4-ddd85f6c7391"). InnerVolumeSpecName "kube-api-access-gr466". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:04:58.197548 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.196745    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b8b93e-f902-4396-bfd4-ddd85f6c7391-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:58.197548 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.196780    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr466\" (UniqueName: \"kubernetes.io/projected/49b8b93e-f902-4396-bfd4-ddd85f6c7391-kube-api-access-gr466\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:58.231288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.230962    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:04:58.248728 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.248615    4872 generic.go:356] "Generic (PLEG): container finished" podID="49b8b93e-f902-4396-bfd4-ddd85f6c7391" containerID="4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9" exitCode=0
Nov 01 21:04:58.248900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.248732    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"
Nov 01 21:04:58.248900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.248841    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm" event={"ID":"49b8b93e-f902-4396-bfd4-ddd85f6c7391","Type":"ContainerDied","Data":"4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9"}
Nov 01 21:04:58.248900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.248893    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm" event={"ID":"49b8b93e-f902-4396-bfd4-ddd85f6c7391","Type":"ContainerDied","Data":"faa9f0269e2f4e9326b637409236a3729f3e726cfa0bbfe4736aef3f3261a590"}
Nov 01 21:04:58.249088 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.248924    4872 scope.go:122] "RemoveContainer" containerID="4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9"
Nov 01 21:04:58.251085 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.251056    4872 generic.go:356] "Generic (PLEG): container finished" podID="6a4867fd-e94a-4159-b66d-7c900001273a" containerID="c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e" exitCode=0
Nov 01 21:04:58.252469 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.252430    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"
Nov 01 21:04:58.252588 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.252531    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx" event={"ID":"6a4867fd-e94a-4159-b66d-7c900001273a","Type":"ContainerDied","Data":"c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e"}
Nov 01 21:04:58.252588 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.252559    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx" event={"ID":"6a4867fd-e94a-4159-b66d-7c900001273a","Type":"ContainerDied","Data":"2dffe43d33e1b3381577f647dd347fc6e95e76a492343e26c7030e15908eab7e"}
Nov 01 21:04:58.273507 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.273481    4872 scope.go:122] "RemoveContainer" containerID="4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9"
Nov 01 21:04:58.273929 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:04:58.273897    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9\": container with ID starting with 4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9 not found: ID does not exist" containerID="4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9"
Nov 01 21:04:58.274030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.273941    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9"} err="failed to get container status \"4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9\": rpc error: code = NotFound desc = could not find container \"4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9\": container with ID starting with 4035ecf2343656c2a735c5cdcf20fddec09b730d03472235890550c977ccf7a9 not found: ID does not exist"
Nov 01 21:04:58.274030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.273965    4872 scope.go:122] "RemoveContainer" containerID="c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e"
Nov 01 21:04:58.284197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.284176    4872 scope.go:122] "RemoveContainer" containerID="c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e"
Nov 01 21:04:58.284514 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:04:58.284491    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e\": container with ID starting with c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e not found: ID does not exist" containerID="c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e"
Nov 01 21:04:58.284600 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.284522    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e"} err="failed to get container status \"c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e\": rpc error: code = NotFound desc = could not find container \"c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e\": container with ID starting with c79cc2b0efa79e5ab835fe28ebe0a3a7d3d64ea005cf9ce54c7400aeb7a6ba8e not found: ID does not exist"
Nov 01 21:04:58.298216 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.298191    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj\" (UniqueName: \"kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj\") pod \"6a4867fd-e94a-4159-b66d-7c900001273a\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") "
Nov 01 21:04:58.298341 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.298240    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume\") pod \"6a4867fd-e94a-4159-b66d-7c900001273a\" (UID: \"6a4867fd-e94a-4159-b66d-7c900001273a\") "
Nov 01 21:04:58.298614 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.298580    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume" pod "6a4867fd-e94a-4159-b66d-7c900001273a" (UID: "6a4867fd-e94a-4159-b66d-7c900001273a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:04:58.298839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.298819    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4867fd-e94a-4159-b66d-7c900001273a-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:58.300318 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.300297    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj" pod "6a4867fd-e94a-4159-b66d-7c900001273a" (UID: "6a4867fd-e94a-4159-b66d-7c900001273a"). InnerVolumeSpecName "kube-api-access-htlqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:04:58.400145 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.400109    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlqj\" (UniqueName: \"kubernetes.io/projected/6a4867fd-e94a-4159-b66d-7c900001273a-kube-api-access-htlqj\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:58.461197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.461150    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"]
Nov 01 21:04:58.465627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.465597    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-081c8e-86f4657b7-wc4tm"]
Nov 01 21:04:58.676222 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.676190    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"]
Nov 01 21:04:58.679872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:58.679844    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-081c8e-9fdd659b7-8w2gx"]
Nov 01 21:04:59.116135 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:59.116108    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b8b93e-f902-4396-bfd4-ddd85f6c7391" path="/var/lib/kubelet/pods/49b8b93e-f902-4396-bfd4-ddd85f6c7391/volumes"
Nov 01 21:04:59.118310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:04:59.118282    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4867fd-e94a-4159-b66d-7c900001273a" path="/var/lib/kubelet/pods/6a4867fd-e94a-4159-b66d-7c900001273a/volumes"
Nov 01 21:05:09.866932 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:09.866900    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"]
Nov 01 21:05:09.867358 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:09.867128    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" podUID="c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" containerName="maas-api" containerID="cri-o://cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2" gracePeriod=30
Nov 01 21:05:10.185125 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.185096    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:05:10.289671 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.289615    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\") pod \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") "
Nov 01 21:05:10.290053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.290030    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn\" (UniqueName: \"kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn\") pod \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") "
Nov 01 21:05:10.290176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.290073    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca\") pod \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") "
Nov 01 21:05:10.290176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.290164    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs\") pod \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\" (UID: \"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a\") "
Nov 01 21:05:10.299579 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.299544    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls" pod "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" (UID: "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:05:10.311675 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.311087    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca" pod "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" (UID: "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:05:10.323825 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.320837    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn" pod "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" (UID: "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a"). InnerVolumeSpecName "kube-api-access-94pjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:05:10.330338 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.324329    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs" pod "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" (UID: "c2858fc5-b3f7-43e9-bfb0-392cabc19a3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:05:10.334944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.334909    4872 generic.go:356] "Generic (PLEG): container finished" podID="c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" containerID="cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2" exitCode=0
Nov 01 21:05:10.335089 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.335068    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" event={"ID":"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a","Type":"ContainerDied","Data":"cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2"}
Nov 01 21:05:10.335164 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.335141    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb" event={"ID":"c2858fc5-b3f7-43e9-bfb0-392cabc19a3a","Type":"ContainerDied","Data":"c05056118ffae06437458ea1f68d482625b36b1541d47a68a64fc4f0ca31e992"}
Nov 01 21:05:10.335221 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.335182    4872 scope.go:122] "RemoveContainer" containerID="cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2"
Nov 01 21:05:10.335368 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.335350    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"
Nov 01 21:05:10.350529 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.350504    4872 scope.go:122] "RemoveContainer" containerID="cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2"
Nov 01 21:05:10.350852 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:05:10.350825    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2\": container with ID starting with cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2 not found: ID does not exist" containerID="cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2"
Nov 01 21:05:10.350960 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.350862    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2"} err="failed to get container status \"cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2\": rpc error: code = NotFound desc = could not find container \"cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2\": container with ID starting with cbb423149eb1b6a201ff44082438358fb20ce48a6b9dda911c95528160dd08f2 not found: ID does not exist"
Nov 01 21:05:10.392725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.392690    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:10.392725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.392724    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pjn\" (UniqueName: \"kubernetes.io/projected/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-kube-api-access-94pjn\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:10.392725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.392735    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:10.392925 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.392745    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:10.503298 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.503218    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"]
Nov 01 21:05:10.508185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:10.508155    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-dab736-5d65f77f9b-jmswb"]
Nov 01 21:05:11.079947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:11.079917    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" path="/var/lib/kubelet/pods/c2858fc5-b3f7-43e9-bfb0-392cabc19a3a/volumes"
Nov 01 21:05:16.033181 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:16.033142    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"]
Nov 01 21:05:23.366322 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:05:23.366287    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:05:29.276038 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:29.276004    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"]
Nov 01 21:05:29.387090 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:29.387057    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"]
Nov 01 21:05:31.058679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.055434    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" containerID="cri-o://8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc" gracePeriod=45
Nov 01 21:05:31.084327 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.084280    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused"
Nov 01 21:05:31.338250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.338225    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 21:05:31.449601 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.449574    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8\" (UniqueName: \"kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8\") pod \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") "
Nov 01 21:05:31.449789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.449677    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs\") pod \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") "
Nov 01 21:05:31.449789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.449731    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache\") pod \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") "
Nov 01 21:05:31.449789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.449773    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir\") pod \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") "
Nov 01 21:05:31.449984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.449809    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm\") pod \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") "
Nov 01 21:05:31.449984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.449837    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home\") pod \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\" (UID: \"25b3d006-8a3e-4506-8d4d-b652f1cf50fa\") "
Nov 01 21:05:31.450099 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.450064    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir" pod "25b3d006-8a3e-4506-8d4d-b652f1cf50fa" (UID: "25b3d006-8a3e-4506-8d4d-b652f1cf50fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:05:31.450161 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.450072    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache" pod "25b3d006-8a3e-4506-8d4d-b652f1cf50fa" (UID: "25b3d006-8a3e-4506-8d4d-b652f1cf50fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:05:31.450161 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.450148    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home" pod "25b3d006-8a3e-4506-8d4d-b652f1cf50fa" (UID: "25b3d006-8a3e-4506-8d4d-b652f1cf50fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:05:31.450358 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.450339    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:31.450423 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.450363    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:31.450423 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.450377    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:31.451928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.451903    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm" pod "25b3d006-8a3e-4506-8d4d-b652f1cf50fa" (UID: "25b3d006-8a3e-4506-8d4d-b652f1cf50fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:05:31.452056 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.451961    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs" pod "25b3d006-8a3e-4506-8d4d-b652f1cf50fa" (UID: "25b3d006-8a3e-4506-8d4d-b652f1cf50fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:05:31.452056 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.451976    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8" pod "25b3d006-8a3e-4506-8d4d-b652f1cf50fa" (UID: "25b3d006-8a3e-4506-8d4d-b652f1cf50fa"). InnerVolumeSpecName "kube-api-access-v6nq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:05:31.512596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.512562    4872 generic.go:356] "Generic (PLEG): container finished" podID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerID="8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc" exitCode=0
Nov 01 21:05:31.512773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.512700    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" event={"ID":"25b3d006-8a3e-4506-8d4d-b652f1cf50fa","Type":"ContainerDied","Data":"8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc"}
Nov 01 21:05:31.512773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.512717    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"
Nov 01 21:05:31.512773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.512739    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn" event={"ID":"25b3d006-8a3e-4506-8d4d-b652f1cf50fa","Type":"ContainerDied","Data":"9e4b302b0a1cf7545f411299399e53ca2df7de59db3e1f235a97c7df7c2b0022"}
Nov 01 21:05:31.512773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.512759    4872 scope.go:122] "RemoveContainer" containerID="8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc"
Nov 01 21:05:31.525219 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.525136    4872 scope.go:122] "RemoveContainer" containerID="8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc"
Nov 01 21:05:31.525457 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:05:31.525435    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc\": container with ID starting with 8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc not found: ID does not exist" containerID="8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc"
Nov 01 21:05:31.525540 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.525467    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc"} err="failed to get container status \"8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc\": rpc error: code = NotFound desc = could not find container \"8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc\": container with ID starting with 8ff494660ddedcbd851779264fbc0d428234ed9baf43865940f769198685b4dc not found: ID does not exist"
Nov 01 21:05:31.552494 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.552468    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:31.552621 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.552500    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6nq8\" (UniqueName: \"kubernetes.io/projected/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-kube-api-access-v6nq8\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:31.552621 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.552515    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3d006-8a3e-4506-8d4d-b652f1cf50fa-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:31.680521 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.680493    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"]
Nov 01 21:05:31.684500 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:31.684475    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/premium-w5-c9ae21-kserve-c5857479b-x6smn"]
Nov 01 21:05:33.080707 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:33.080637    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" path="/var/lib/kubelet/pods/25b3d006-8a3e-4506-8d4d-b652f1cf50fa/volumes"
Nov 01 21:05:34.276922 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.276867    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd" podUID="0198310b-397b-4a13-b99a-6783d85f140d" containerName="payload-processing" containerID="cri-o://c7a351427ea1aacf04f13d3983adcaaeb1ec6a7983cdbeaf30269f616675b911" gracePeriod=25
Nov 01 21:05:34.388267 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.388232    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7" podUID="936bd6c6-10e6-4d18-8739-64d67c6610c3" containerName="payload-pre-processing" containerID="cri-o://b989a50a6fe7ee31fee6a8268b410d9e668abb7110c4d4892a3a5ddcfb1eab94" gracePeriod=25
Nov 01 21:05:34.539057 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.539022    4872 generic.go:356] "Generic (PLEG): container finished" podID="0198310b-397b-4a13-b99a-6783d85f140d" containerID="c7a351427ea1aacf04f13d3983adcaaeb1ec6a7983cdbeaf30269f616675b911" exitCode=0
Nov 01 21:05:34.539185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.539101    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd" event={"ID":"0198310b-397b-4a13-b99a-6783d85f140d","Type":"ContainerDied","Data":"c7a351427ea1aacf04f13d3983adcaaeb1ec6a7983cdbeaf30269f616675b911"}
Nov 01 21:05:34.542188 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.542067    4872 generic.go:356] "Generic (PLEG): container finished" podID="936bd6c6-10e6-4d18-8739-64d67c6610c3" containerID="b989a50a6fe7ee31fee6a8268b410d9e668abb7110c4d4892a3a5ddcfb1eab94" exitCode=0
Nov 01 21:05:34.542188 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.542140    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7" event={"ID":"936bd6c6-10e6-4d18-8739-64d67c6610c3","Type":"ContainerDied","Data":"b989a50a6fe7ee31fee6a8268b410d9e668abb7110c4d4892a3a5ddcfb1eab94"}
Nov 01 21:05:34.562842 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.562818    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:05:34.692141 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.692115    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:05:34.696792 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.696768    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume\") pod \"936bd6c6-10e6-4d18-8739-64d67c6610c3\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") "
Nov 01 21:05:34.696894 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.696850    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55\" (UniqueName: \"kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55\") pod \"0198310b-397b-4a13-b99a-6783d85f140d\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") "
Nov 01 21:05:34.696948 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.696895    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume\") pod \"0198310b-397b-4a13-b99a-6783d85f140d\" (UID: \"0198310b-397b-4a13-b99a-6783d85f140d\") "
Nov 01 21:05:34.697004 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.696948    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq\" (UniqueName: \"kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq\") pod \"936bd6c6-10e6-4d18-8739-64d67c6610c3\" (UID: \"936bd6c6-10e6-4d18-8739-64d67c6610c3\") "
Nov 01 21:05:34.697220 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.697175    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume" pod "936bd6c6-10e6-4d18-8739-64d67c6610c3" (UID: "936bd6c6-10e6-4d18-8739-64d67c6610c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:05:34.697549 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.697529    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume" pod "0198310b-397b-4a13-b99a-6783d85f140d" (UID: "0198310b-397b-4a13-b99a-6783d85f140d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:05:34.697807 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.697785    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bd6c6-10e6-4d18-8739-64d67c6610c3-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:34.697873 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.697814    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0198310b-397b-4a13-b99a-6783d85f140d-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:34.699516 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.699496    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55" pod "0198310b-397b-4a13-b99a-6783d85f140d" (UID: "0198310b-397b-4a13-b99a-6783d85f140d"). InnerVolumeSpecName "kube-api-access-hxt55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:05:34.699589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.699521    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq" pod "936bd6c6-10e6-4d18-8739-64d67c6610c3" (UID: "936bd6c6-10e6-4d18-8739-64d67c6610c3"). InnerVolumeSpecName "kube-api-access-df9tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:05:34.799539 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.799486    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxt55\" (UniqueName: \"kubernetes.io/projected/0198310b-397b-4a13-b99a-6783d85f140d-kube-api-access-hxt55\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:34.799539 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:34.799510    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9tq\" (UniqueName: \"kubernetes.io/projected/936bd6c6-10e6-4d18-8739-64d67c6610c3-kube-api-access-df9tq\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:35.549498 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.549467    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"
Nov 01 21:05:35.549994 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.549505    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7" event={"ID":"936bd6c6-10e6-4d18-8739-64d67c6610c3","Type":"ContainerDied","Data":"2746b72fd753f8fba60eb6212ef44c39e8e007e962487a95ed92952865144449"}
Nov 01 21:05:35.549994 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.549560    4872 scope.go:122] "RemoveContainer" containerID="b989a50a6fe7ee31fee6a8268b410d9e668abb7110c4d4892a3a5ddcfb1eab94"
Nov 01 21:05:35.551100 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.551075    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"
Nov 01 21:05:35.551198 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.551101    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd" event={"ID":"0198310b-397b-4a13-b99a-6783d85f140d","Type":"ContainerDied","Data":"5a1b85d31150a6aac962b244268041d5d2a1b3f22dd6ecb955d276aa55c425bc"}
Nov 01 21:05:35.563213 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.563183    4872 scope.go:122] "RemoveContainer" containerID="c7a351427ea1aacf04f13d3983adcaaeb1ec6a7983cdbeaf30269f616675b911"
Nov 01 21:05:35.715431 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.715410    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"]
Nov 01 21:05:35.765846 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.765823    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-11a069b1-7c85476fd5-mlmjd"]
Nov 01 21:05:35.827986 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.827920    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"]
Nov 01 21:05:35.831243 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:35.831223    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-11a069b1-6977f77fd7-lvtg7"]
Nov 01 21:05:37.079328 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:37.079298    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0198310b-397b-4a13-b99a-6783d85f140d" path="/var/lib/kubelet/pods/0198310b-397b-4a13-b99a-6783d85f140d/volumes"
Nov 01 21:05:37.079732 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:37.079717    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936bd6c6-10e6-4d18-8739-64d67c6610c3" path="/var/lib/kubelet/pods/936bd6c6-10e6-4d18-8739-64d67c6610c3/volumes"
Nov 01 21:05:43.481160 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:43.481132    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:05:43.483058 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:43.483034    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:05:43.487295 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:43.487272    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:05:43.489060 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:43.489040    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:05:53.367192 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:05:53.367151    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:05:55.949032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:55.948952    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"]
Nov 01 21:05:55.949450 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:05:55.949201    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy" containerID="cri-o://997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892" gracePeriod=30
Nov 01 21:06:01.229031 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.228975    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:06:01.330239 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330213    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330261    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330307    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330344    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330555 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330413    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330555 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330440    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330553    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330604    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330782 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330754    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket\") pod \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\" (UID: \"cf20bb71-b131-43ca-bd74-0e3ad32a05ed\") "
Nov 01 21:06:01.330836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.330799    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:01.331042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331021    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:01.331144 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331117    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:06:01.331267 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331246    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:01.331342 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331267    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:01.331492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331462    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.331492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331489    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.331584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331504    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.331584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331518    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.331584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.331531    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.332598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.332573    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:06:01.332995 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.332971    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:06:01.333098 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.333041    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "kube-api-access-brcb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:06:01.333098 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.333080    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy" pod "cf20bb71-b131-43ca-bd74-0e3ad32a05ed" (UID: "cf20bb71-b131-43ca-bd74-0e3ad32a05ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:01.433699 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.433651    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.433893 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.433703    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brcb8\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-kube-api-access-brcb8\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.433893 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.433720    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.433893 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.433735    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf20bb71-b131-43ca-bd74-0e3ad32a05ed-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:01.740299 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.740266    4872 generic.go:356] "Generic (PLEG): container finished" podID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerID="997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892" exitCode=0
Nov 01 21:06:01.740465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.740310    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"
Nov 01 21:06:01.740465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.740344    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" event={"ID":"cf20bb71-b131-43ca-bd74-0e3ad32a05ed","Type":"ContainerDied","Data":"997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892"}
Nov 01 21:06:01.740465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.740394    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq" event={"ID":"cf20bb71-b131-43ca-bd74-0e3ad32a05ed","Type":"ContainerDied","Data":"3992edc573f0c927c330f4338977c26bc60d56db5ab3a905c186a92d17a46e22"}
Nov 01 21:06:01.740465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.740416    4872 scope.go:122] "RemoveContainer" containerID="997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892"
Nov 01 21:06:01.751076 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.751062    4872 scope.go:122] "RemoveContainer" containerID="3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"
Nov 01 21:06:01.759740 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.759724    4872 scope.go:122] "RemoveContainer" containerID="997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892"
Nov 01 21:06:01.759992 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:06:01.759973    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892\": container with ID starting with 997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892 not found: ID does not exist" containerID="997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892"
Nov 01 21:06:01.760032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.760001    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892"} err="failed to get container status \"997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892\": rpc error: code = NotFound desc = could not find container \"997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892\": container with ID starting with 997d063267e0816c74964affc1cfd0aa07c442021dd3e98c9347d97892bac892 not found: ID does not exist"
Nov 01 21:06:01.760032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.760018    4872 scope.go:122] "RemoveContainer" containerID="3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"
Nov 01 21:06:01.760233 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:06:01.760216    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b\": container with ID starting with 3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b not found: ID does not exist" containerID="3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"
Nov 01 21:06:01.760266 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.760239    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b"} err="failed to get container status \"3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b\": rpc error: code = NotFound desc = could not find container \"3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b\": container with ID starting with 3b5ea6d1c1f7c759c8cc08dca61836484e348b6543cd3003589c0a41cc9aa37b not found: ID does not exist"
Nov 01 21:06:01.908041 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.908009    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"]
Nov 01 21:06:01.958530 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:01.958508    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dd840a-openshift-default-54cd84677f-dvjrq"]
Nov 01 21:06:03.081801 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:03.081758    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" path="/var/lib/kubelet/pods/cf20bb71-b131-43ca-bd74-0e3ad32a05ed/volumes"
Nov 01 21:06:14.962620 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:14.962590    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"]
Nov 01 21:06:14.962986 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:14.962812    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" podUID="4168885a-d9b3-473e-abd5-f283b05a5de0" containerName="maas-api" containerID="cri-o://bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800" gracePeriod=30
Nov 01 21:06:15.279943 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.279919    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:06:15.422381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.421632    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca\") pod \"4168885a-d9b3-473e-abd5-f283b05a5de0\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") "
Nov 01 21:06:15.422381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.421819    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs\") pod \"4168885a-d9b3-473e-abd5-f283b05a5de0\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") "
Nov 01 21:06:15.422381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.421904    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg\" (UniqueName: \"kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg\") pod \"4168885a-d9b3-473e-abd5-f283b05a5de0\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") "
Nov 01 21:06:15.422381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.422016    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\") pod \"4168885a-d9b3-473e-abd5-f283b05a5de0\" (UID: \"4168885a-d9b3-473e-abd5-f283b05a5de0\") "
Nov 01 21:06:15.425024 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.424998    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs" pod "4168885a-d9b3-473e-abd5-f283b05a5de0" (UID: "4168885a-d9b3-473e-abd5-f283b05a5de0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:06:15.425333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.425307    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg" pod "4168885a-d9b3-473e-abd5-f283b05a5de0" (UID: "4168885a-d9b3-473e-abd5-f283b05a5de0"). InnerVolumeSpecName "kube-api-access-v8mtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:06:15.425410 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.425314    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca" pod "4168885a-d9b3-473e-abd5-f283b05a5de0" (UID: "4168885a-d9b3-473e-abd5-f283b05a5de0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:06:15.426891 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.426863    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls" pod "4168885a-d9b3-473e-abd5-f283b05a5de0" (UID: "4168885a-d9b3-473e-abd5-f283b05a5de0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:06:15.525939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.525849    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4168885a-d9b3-473e-abd5-f283b05a5de0-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:15.525939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.525892    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:15.525939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.525908    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mtg\" (UniqueName: \"kubernetes.io/projected/4168885a-d9b3-473e-abd5-f283b05a5de0-kube-api-access-v8mtg\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:15.525939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.525924    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4168885a-d9b3-473e-abd5-f283b05a5de0-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:15.832527 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.832449    4872 generic.go:356] "Generic (PLEG): container finished" podID="4168885a-d9b3-473e-abd5-f283b05a5de0" containerID="bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800" exitCode=0
Nov 01 21:06:15.832527 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.832493    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"
Nov 01 21:06:15.832778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.832536    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" event={"ID":"4168885a-d9b3-473e-abd5-f283b05a5de0","Type":"ContainerDied","Data":"bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800"}
Nov 01 21:06:15.832778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.832588    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8" event={"ID":"4168885a-d9b3-473e-abd5-f283b05a5de0","Type":"ContainerDied","Data":"c681d0e9a18f74e171b16228320bcbf01b39559c5bd7d1981fd999a70ddbbdf5"}
Nov 01 21:06:15.832778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.832611    4872 scope.go:122] "RemoveContainer" containerID="bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800"
Nov 01 21:06:15.843403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.843384    4872 scope.go:122] "RemoveContainer" containerID="bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800"
Nov 01 21:06:15.843646 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:06:15.843630    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800\": container with ID starting with bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800 not found: ID does not exist" containerID="bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800"
Nov 01 21:06:15.843785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:15.843678    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800"} err="failed to get container status \"bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800\": rpc error: code = NotFound desc = could not find container \"bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800\": container with ID starting with bbd54ca1c64419faaa13e0bbd05c504c7a02264733860f44e95474851ae28800 not found: ID does not exist"
Nov 01 21:06:16.009227 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:16.009187    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"]
Nov 01 21:06:16.014923 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:16.014898    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-22ed7b-6bdcdfc85c-h9hg8"]
Nov 01 21:06:17.082484 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:17.082433    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4168885a-d9b3-473e-abd5-f283b05a5de0" path="/var/lib/kubelet/pods/4168885a-d9b3-473e-abd5-f283b05a5de0/volumes"
Nov 01 21:06:23.367442 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:06:23.367402    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:06:31.787586 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:31.787547    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"]
Nov 01 21:06:31.788045 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:31.787870    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" podUID="275ee54a-e089-40ed-96e3-4d2251b8c658" containerName="maas-api" containerID="cri-o://4f776ea95a15fb1bc0fcd6cbccae3c5aa111b34d0a59e94d9bc85f90ec9448b2" gracePeriod=30
Nov 01 21:06:31.955998 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:31.955961    4872 generic.go:356] "Generic (PLEG): container finished" podID="275ee54a-e089-40ed-96e3-4d2251b8c658" containerID="4f776ea95a15fb1bc0fcd6cbccae3c5aa111b34d0a59e94d9bc85f90ec9448b2" exitCode=0
Nov 01 21:06:31.956205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:31.956035    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" event={"ID":"275ee54a-e089-40ed-96e3-4d2251b8c658","Type":"ContainerDied","Data":"4f776ea95a15fb1bc0fcd6cbccae3c5aa111b34d0a59e94d9bc85f90ec9448b2"}
Nov 01 21:06:32.102120 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.102094    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:06:32.226712 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.226674    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca\") pod \"275ee54a-e089-40ed-96e3-4d2251b8c658\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") "
Nov 01 21:06:32.226936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.226805    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs\") pod \"275ee54a-e089-40ed-96e3-4d2251b8c658\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") "
Nov 01 21:06:32.226936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.226884    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd\" (UniqueName: \"kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd\") pod \"275ee54a-e089-40ed-96e3-4d2251b8c658\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") "
Nov 01 21:06:32.227089 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.227011    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\") pod \"275ee54a-e089-40ed-96e3-4d2251b8c658\" (UID: \"275ee54a-e089-40ed-96e3-4d2251b8c658\") "
Nov 01 21:06:32.227089 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.227051    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca" pod "275ee54a-e089-40ed-96e3-4d2251b8c658" (UID: "275ee54a-e089-40ed-96e3-4d2251b8c658"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:06:32.227639 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.227613    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275ee54a-e089-40ed-96e3-4d2251b8c658-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:32.229593 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.229567    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs" pod "275ee54a-e089-40ed-96e3-4d2251b8c658" (UID: "275ee54a-e089-40ed-96e3-4d2251b8c658"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:06:32.230056 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.230031    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd" pod "275ee54a-e089-40ed-96e3-4d2251b8c658" (UID: "275ee54a-e089-40ed-96e3-4d2251b8c658"). InnerVolumeSpecName "kube-api-access-jkpcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:06:32.230959 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.230934    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls" pod "275ee54a-e089-40ed-96e3-4d2251b8c658" (UID: "275ee54a-e089-40ed-96e3-4d2251b8c658"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:06:32.328736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.328630    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:32.328736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.328675    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275ee54a-e089-40ed-96e3-4d2251b8c658-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:32.328736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.328686    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpcd\" (UniqueName: \"kubernetes.io/projected/275ee54a-e089-40ed-96e3-4d2251b8c658-kube-api-access-jkpcd\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:32.966574 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.966544    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"
Nov 01 21:06:32.967011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.966546    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg" event={"ID":"275ee54a-e089-40ed-96e3-4d2251b8c658","Type":"ContainerDied","Data":"08f57b59147139a22f75f18e0bb534d228dba1b1fe264a3549ecd24eb5894d63"}
Nov 01 21:06:32.967011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:32.966699    4872 scope.go:122] "RemoveContainer" containerID="4f776ea95a15fb1bc0fcd6cbccae3c5aa111b34d0a59e94d9bc85f90ec9448b2"
Nov 01 21:06:33.133177 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:33.133141    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"]
Nov 01 21:06:33.183821 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:33.183788    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11a069b1-675bb68865-t7xcg"]
Nov 01 21:06:35.084726 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:35.084688    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275ee54a-e089-40ed-96e3-4d2251b8c658" path="/var/lib/kubelet/pods/275ee54a-e089-40ed-96e3-4d2251b8c658/volumes"
Nov 01 21:06:40.578714 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:40.578640    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"]
Nov 01 21:06:53.367116 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:06:53.367070    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:06:55.600469 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:55.600419    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" podUID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" containerName="main" containerID="cri-o://9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82" gracePeriod=45
Nov 01 21:06:55.747972 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:55.747925    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" podUID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": dial tcp 10.132.0.104:8000: connect: connection refused"
Nov 01 21:06:55.865943 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:55.865872    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 21:06:56.005709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.005640    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir\") pod \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") "
Nov 01 21:06:56.005918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.005814    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm\") pod \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") "
Nov 01 21:06:56.005918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.005892    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr\" (UniqueName: \"kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr\") pod \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") "
Nov 01 21:06:56.006032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.005921    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache\") pod \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") "
Nov 01 21:06:56.006086 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.006052    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home\") pod \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") "
Nov 01 21:06:56.006205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.006091    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs\" (UniqueName: \"kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs\") pod \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\" (UID: \"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28\") "
Nov 01 21:06:56.006205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.005918    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir" pod "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" (UID: "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:56.006357 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.006336    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home" pod "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" (UID: "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:56.006717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.006696    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:56.006885 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.006869    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:56.006980 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.006717    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache" pod "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" (UID: "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:56.008968 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.008931    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm" pod "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" (UID: "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:06:56.009079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.008972    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr" pod "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" (UID: "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28"). InnerVolumeSpecName "kube-api-access-jvrfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:06:56.009234 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.009200    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs" pod "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" (UID: "279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:06:56.108173 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.108134    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:56.108173 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.108179    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:56.108390 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.108195    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrfr\" (UniqueName: \"kubernetes.io/projected/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-kube-api-access-jvrfr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:56.108390 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.108209    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:06:56.126712 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.126610    4872 generic.go:356] "Generic (PLEG): container finished" podID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" containerID="9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82" exitCode=0
Nov 01 21:06:56.126712 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.126664    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"
Nov 01 21:06:56.126712 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.126695    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" event={"ID":"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28","Type":"ContainerDied","Data":"9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82"}
Nov 01 21:06:56.126973 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.126741    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp" event={"ID":"279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28","Type":"ContainerDied","Data":"c8745a0fb63cb9e4172800de111e2e14b22bc34e77af342570b02cf27ba668a3"}
Nov 01 21:06:56.126973 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.126764    4872 scope.go:122] "RemoveContainer" containerID="9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82"
Nov 01 21:06:56.137295 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.137278    4872 scope.go:122] "RemoveContainer" containerID="9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82"
Nov 01 21:06:56.137537 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:06:56.137521    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82\": container with ID starting with 9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82 not found: ID does not exist" containerID="9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82"
Nov 01 21:06:56.137584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.137545    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82"} err="failed to get container status \"9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82\": rpc error: code = NotFound desc = could not find container \"9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82\": container with ID starting with 9d28aecd1a1a814187abd35771ce3061d4e865bea1cfc320a2503d578eaefc82 not found: ID does not exist"
Nov 01 21:06:56.293436 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.293395    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"]
Nov 01 21:06:56.299105 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:56.299074    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-11a069b1/e2e-lifecycle-model-11a069b1-kserve-85cdb9687c-4s7qp"]
Nov 01 21:06:57.079927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:57.079893    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" path="/var/lib/kubelet/pods/279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28/volumes"
Nov 01 21:06:59.762629 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:06:59.762591    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"]
Nov 01 21:07:04.763819 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:04.763774    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5" podUID="c9688de2-bf9f-429e-a098-660232194e5b" containerName="payload-processing" containerID="cri-o://d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00" gracePeriod=25
Nov 01 21:07:05.059602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.059569    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:07:05.125110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.125073    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume\") pod \"c9688de2-bf9f-429e-a098-660232194e5b\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") "
Nov 01 21:07:05.125324 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.125259    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn\" (UniqueName: \"kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn\") pod \"c9688de2-bf9f-429e-a098-660232194e5b\" (UID: \"c9688de2-bf9f-429e-a098-660232194e5b\") "
Nov 01 21:07:05.125558 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.125527    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume" pod "c9688de2-bf9f-429e-a098-660232194e5b" (UID: "c9688de2-bf9f-429e-a098-660232194e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:07:05.125936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.125915    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9688de2-bf9f-429e-a098-660232194e5b-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:05.127352 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.127331    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn" pod "c9688de2-bf9f-429e-a098-660232194e5b" (UID: "c9688de2-bf9f-429e-a098-660232194e5b"). InnerVolumeSpecName "kube-api-access-fvlkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:07:05.194118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.194077    4872 generic.go:356] "Generic (PLEG): container finished" podID="c9688de2-bf9f-429e-a098-660232194e5b" containerID="d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00" exitCode=0
Nov 01 21:07:05.194118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.194130    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"
Nov 01 21:07:05.194392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.194174    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5" event={"ID":"c9688de2-bf9f-429e-a098-660232194e5b","Type":"ContainerDied","Data":"d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00"}
Nov 01 21:07:05.194392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.194236    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5" event={"ID":"c9688de2-bf9f-429e-a098-660232194e5b","Type":"ContainerDied","Data":"9e583cb89d0a51e14811447827934301133727852d5dc96d349bfbad9be88cf8"}
Nov 01 21:07:05.194392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.194269    4872 scope.go:122] "RemoveContainer" containerID="d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00"
Nov 01 21:07:05.207782 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.207762    4872 scope.go:122] "RemoveContainer" containerID="d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00"
Nov 01 21:07:05.213623 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:05.213592    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00\": container with ID starting with d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00 not found: ID does not exist" containerID="d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00"
Nov 01 21:07:05.213788 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.213637    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00"} err="failed to get container status \"d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00\": rpc error: code = NotFound desc = could not find container \"d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00\": container with ID starting with d4b55da07ff2dda44654af953f9b9e90285f39ce167f23610c8583e014bd6b00 not found: ID does not exist"
Nov 01 21:07:05.228339 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.228289    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlkn\" (UniqueName: \"kubernetes.io/projected/c9688de2-bf9f-429e-a098-660232194e5b-kube-api-access-fvlkn\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:05.361232 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.361140    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"]
Nov 01 21:07:05.364198 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:05.364173    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d08fd0-7c95697f8f-sxbm5"]
Nov 01 21:07:07.079620 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:07.079576    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9688de2-bf9f-429e-a098-660232194e5b" path="/var/lib/kubelet/pods/c9688de2-bf9f-429e-a098-660232194e5b/volumes"
Nov 01 21:07:23.366990 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:23.366945    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:07:23.786647 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:23.786617    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"]
Nov 01 21:07:23.786870 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:23.786845    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" containerID="cri-o://e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4" gracePeriod=30
Nov 01 21:07:23.791193 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:23.791167    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"]
Nov 01 21:07:23.791444 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:23.791406    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" containerID="cri-o://15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1" gracePeriod=30
Nov 01 21:07:26.940775 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:26.940697    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:07:26.940775 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:26.940760    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:07:27.970934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:27.970898    4872 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:07:27.971325 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:27.970959    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:07:29.078285 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.078259    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:07:29.115427 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.115400    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:07:29.152760 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152737    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.152856 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152774    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.152856 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152831    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.152966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152866    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.152966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152889    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.152966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152916    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.153121 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152973    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.153121 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.152998    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153121 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153013    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153123    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153164    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153150    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153217    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153250    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153250    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153256    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.153286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153256    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153380    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153398    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153410    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153469    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153484    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153538    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153590    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153605    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token\") pod \"2685675b-088e-4fd8-80d0-373afccd70d0\" (UID: \"2685675b-088e-4fd8-80d0-373afccd70d0\") "
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153608    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:07:29.153733 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.153720    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket\") pod \"835b0743-70c1-4021-9655-ac6faadd38f9\" (UID: \"835b0743-70c1-4021-9655-ac6faadd38f9\") "
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154327    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154352    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835b0743-70c1-4021-9655-ac6faadd38f9-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154367    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154383    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154399    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154414    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154427    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685675b-088e-4fd8-80d0-373afccd70d0-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154442    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154456    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.154613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.154580    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.155589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.155564    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "kube-api-access-hg99n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:07:29.156044 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.155931    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.156044 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.156019    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "kube-api-access-5hbq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:07:29.156189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.156123    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:07:29.156299 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.156200    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:07:29.156447 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.156424    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token" pod "2685675b-088e-4fd8-80d0-373afccd70d0" (UID: "2685675b-088e-4fd8-80d0-373afccd70d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:07:29.156618 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.156592    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:07:29.157223 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.157197    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo" pod "835b0743-70c1-4021-9655-ac6faadd38f9" (UID: "835b0743-70c1-4021-9655-ac6faadd38f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:07:29.256126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256099    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685675b-088e-4fd8-80d0-373afccd70d0-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256126    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-kube-api-access-5hbq2\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256139    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685675b-088e-4fd8-80d0-373afccd70d0-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256152    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256164    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685675b-088e-4fd8-80d0-373afccd70d0-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256176    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256190    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg99n\" (UniqueName: \"kubernetes.io/projected/835b0743-70c1-4021-9655-ac6faadd38f9-kube-api-access-hg99n\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256202    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835b0743-70c1-4021-9655-ac6faadd38f9-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.256294 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.256215    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835b0743-70c1-4021-9655-ac6faadd38f9-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:07:29.367185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.367099    4872 generic.go:356] "Generic (PLEG): container finished" podID="2685675b-088e-4fd8-80d0-373afccd70d0" containerID="15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1" exitCode=0
Nov 01 21:07:29.367185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.367140    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"
Nov 01 21:07:29.367408 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.367187    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" event={"ID":"2685675b-088e-4fd8-80d0-373afccd70d0","Type":"ContainerDied","Data":"15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1"}
Nov 01 21:07:29.367408 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.367254    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb" event={"ID":"2685675b-088e-4fd8-80d0-373afccd70d0","Type":"ContainerDied","Data":"4b060d8de828b9d73a758038d03b0ddb3a1f713ce8593b1737702587522142c2"}
Nov 01 21:07:29.367408 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.367285    4872 scope.go:122] "RemoveContainer" containerID="15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1"
Nov 01 21:07:29.369095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.369069    4872 generic.go:356] "Generic (PLEG): container finished" podID="835b0743-70c1-4021-9655-ac6faadd38f9" containerID="e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4" exitCode=0
Nov 01 21:07:29.369236 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.369140    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"
Nov 01 21:07:29.369236 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.369148    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" event={"ID":"835b0743-70c1-4021-9655-ac6faadd38f9","Type":"ContainerDied","Data":"e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4"}
Nov 01 21:07:29.369236 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.369189    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2" event={"ID":"835b0743-70c1-4021-9655-ac6faadd38f9","Type":"ContainerDied","Data":"0d2b438f4ffe8b1ccd7f5a4e0ed130b9813a3602e25de49f6350568a7a8ebaa6"}
Nov 01 21:07:29.380558 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.380540    4872 scope.go:122] "RemoveContainer" containerID="15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1"
Nov 01 21:07:29.381006 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:29.380861    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1\": container with ID starting with 15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1 not found: ID does not exist" containerID="15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1"
Nov 01 21:07:29.381006 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.380901    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1"} err="failed to get container status \"15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1\": rpc error: code = NotFound desc = could not find container \"15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1\": container with ID starting with 15faa753140ea4830f98be455e3408392fbc6e95d1e170fd8274a593f70230f1 not found: ID does not exist"
Nov 01 21:07:29.381006 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.380926    4872 scope.go:122] "RemoveContainer" containerID="e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4"
Nov 01 21:07:29.395041 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.395008    4872 scope.go:122] "RemoveContainer" containerID="e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4"
Nov 01 21:07:29.395337 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:29.395314    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4\": container with ID starting with e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4 not found: ID does not exist" containerID="e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4"
Nov 01 21:07:29.395423 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.395345    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4"} err="failed to get container status \"e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4\": rpc error: code = NotFound desc = could not find container \"e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4\": container with ID starting with e283fcbfe16b537acebad8602f2359c5ccb70ab016624d7dddc19d7b12b66af4 not found: ID does not exist"
Nov 01 21:07:29.526961 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.526870    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"]
Nov 01 21:07:29.530130 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.530103    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-jswz2"]
Nov 01 21:07:29.641033 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.640953    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"]
Nov 01 21:07:29.644364 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:29.644344    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-zqwvb"]
Nov 01 21:07:31.079147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:31.079111    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" path="/var/lib/kubelet/pods/2685675b-088e-4fd8-80d0-373afccd70d0/volumes"
Nov 01 21:07:31.079612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:31.079595    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" path="/var/lib/kubelet/pods/835b0743-70c1-4021-9655-ac6faadd38f9/volumes"
Nov 01 21:07:50.267345 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267309    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"]
Nov 01 21:07:50.267901 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267886    4872 cpu_manager.go:414] "containerMap: removing container" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy"
Nov 01 21:07:50.267947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267902    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy"
Nov 01 21:07:50.267947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267911    4872 cpu_manager.go:414] "containerMap: removing container" podUID="0198310b-397b-4a13-b99a-6783d85f140d" containerName="payload-processing"
Nov 01 21:07:50.267947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267916    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0198310b-397b-4a13-b99a-6783d85f140d" containerName="payload-processing"
Nov 01 21:07:50.267947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267932    4872 cpu_manager.go:414] "containerMap: removing container" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy"
Nov 01 21:07:50.267947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267938    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy"
Nov 01 21:07:50.267947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267945    4872 cpu_manager.go:414] "containerMap: removing container" podUID="936bd6c6-10e6-4d18-8739-64d67c6610c3" containerName="payload-pre-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267950    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936bd6c6-10e6-4d18-8739-64d67c6610c3" containerName="payload-pre-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267962    4872 cpu_manager.go:414] "containerMap: removing container" podUID="4168885a-d9b3-473e-abd5-f283b05a5de0" containerName="maas-api"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267967    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4168885a-d9b3-473e-abd5-f283b05a5de0" containerName="maas-api"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267977    4872 cpu_manager.go:414] "containerMap: removing container" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267981    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267988    4872 cpu_manager.go:414] "containerMap: removing container" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267992    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.267999    4872 cpu_manager.go:414] "containerMap: removing container" podUID="49b8b93e-f902-4396-bfd4-ddd85f6c7391" containerName="payload-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268004    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b8b93e-f902-4396-bfd4-ddd85f6c7391" containerName="payload-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268010    4872 cpu_manager.go:414] "containerMap: removing container" podUID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" containerName="main"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268014    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" containerName="main"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268024    4872 cpu_manager.go:414] "containerMap: removing container" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268029    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268035    4872 cpu_manager.go:414] "containerMap: removing container" podUID="6a4867fd-e94a-4159-b66d-7c900001273a" containerName="payload-pre-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268040    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4867fd-e94a-4159-b66d-7c900001273a" containerName="payload-pre-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268052    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" containerName="maas-api"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268056    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" containerName="maas-api"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268070    4872 cpu_manager.go:414] "containerMap: removing container" podUID="275ee54a-e089-40ed-96e3-4d2251b8c658" containerName="maas-api"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268074    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275ee54a-e089-40ed-96e3-4d2251b8c658" containerName="maas-api"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268087    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c9688de2-bf9f-429e-a098-660232194e5b" containerName="payload-processing"
Nov 01 21:07:50.268132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268091    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9688de2-bf9f-429e-a098-660232194e5b" containerName="payload-processing"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268164    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="275ee54a-e089-40ed-96e3-4d2251b8c658" containerName="maas-api"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268175    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9688de2-bf9f-429e-a098-660232194e5b" containerName="payload-processing"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268181    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268187    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="0198310b-397b-4a13-b99a-6783d85f140d" containerName="payload-processing"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268194    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf20bb71-b131-43ca-bd74-0e3ad32a05ed" containerName="istio-proxy"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268199    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="936bd6c6-10e6-4d18-8739-64d67c6610c3" containerName="payload-pre-processing"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268207    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="4168885a-d9b3-473e-abd5-f283b05a5de0" containerName="maas-api"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268212    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b3d006-8a3e-4506-8d4d-b652f1cf50fa" containerName="main"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268217    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="2685675b-088e-4fd8-80d0-373afccd70d0" containerName="istio-proxy"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268222    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b8b93e-f902-4396-bfd4-ddd85f6c7391" containerName="payload-processing"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268227    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="279f0ac6-d5c0-4962-85fe-d4d0d7c6ec28" containerName="main"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268233    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="835b0743-70c1-4021-9655-ac6faadd38f9" containerName="istio-proxy"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268239    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4867fd-e94a-4159-b66d-7c900001273a" containerName="payload-pre-processing"
Nov 01 21:07:50.268826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.268245    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2858fc5-b3f7-43e9-bfb0-392cabc19a3a" containerName="maas-api"
Nov 01 21:07:50.273231 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.273210    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.327886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.327855    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"]
Nov 01 21:07:50.346859 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.346824    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.347052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.347028    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.347132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.347070    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.347132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.347099    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdntr\" (UniqueName: \"kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.374890 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.374856    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6b9305-serving-cert\""
Nov 01 21:07:50.448954 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.448925    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.449114 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.449079    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.449114 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.449111    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.449197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.449135    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdntr\" (UniqueName: \"kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.449265 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:50.449248    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-serving-cert: secret "maas-api-e2e-inf-a-6b9305-serving-cert" not found
Nov 01 21:07:50.449325 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:50.449315    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls podName:a2ae5315-ccc0-4ac7-bc6b-3510df8b1420 nodeName:}" failed. No retries permitted until 2026-11-01 21:07:50.949296145 +0000 UTC m=+2227.998405956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls") pod "maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" (UID: "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420") : secret "maas-api-e2e-inf-a-6b9305-serving-cert" not found
Nov 01 21:07:50.449770 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.449749    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.451495 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.451473    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.516773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.516739    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdntr\" (UniqueName: \"kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.954850 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.954818    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:50.957272 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:50.957251    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:51.187620 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:51.187590    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:51.566880 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:51.566846    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"]
Nov 01 21:07:51.568120 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:07:51.568092    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ae5315_ccc0_4ac7_bc6b_3510df8b1420.slice/crio-2d814d7dfd5a7c123d441fd81006b5381eb6066c70c5311bdb73f0b2a1bd63e8 WatchSource:0}: Error finding container 2d814d7dfd5a7c123d441fd81006b5381eb6066c70c5311bdb73f0b2a1bd63e8: Status 404 returned error can't find the container with id 2d814d7dfd5a7c123d441fd81006b5381eb6066c70c5311bdb73f0b2a1bd63e8
Nov 01 21:07:51.569484 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:51.569465    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:07:52.515895 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:52.515866    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" event={"ID":"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420","Type":"ContainerStarted","Data":"2d814d7dfd5a7c123d441fd81006b5381eb6066c70c5311bdb73f0b2a1bd63e8"}
Nov 01 21:07:53.401287 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:07:53.401245    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:07:53.526766 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:53.526728    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" event={"ID":"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420","Type":"ContainerStarted","Data":"f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec"}
Nov 01 21:07:53.526964 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:53.526884    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:07:53.644416 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:53.644361    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" podStartSLOduration=2.7544598479999998 podStartE2EDuration="3.644346108s" totalImagesPullingTime="889.88626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:07:53.64123535 +0000 UTC m=+2230.690345172" watchObservedRunningTime="2026-11-01 21:07:53.644346108 +0000 UTC m=+2230.693455936"
Nov 01 21:07:59.539508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:07:59.539466    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:08:04.516078 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:04.516036    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"]
Nov 01 21:08:04.516617 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:04.516301    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy" containerID="cri-o://9c24561ef41ee91bb4a6a2f75bde907e99a399ddef66a1825b02eb6e4e478879" gracePeriod=30
Nov 01 21:08:09.635684 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.635617    4872 generic.go:356] "Generic (PLEG): container finished" podID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerID="9c24561ef41ee91bb4a6a2f75bde907e99a399ddef66a1825b02eb6e4e478879" exitCode=0
Nov 01 21:08:09.636168 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.635750    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" event={"ID":"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7","Type":"ContainerDied","Data":"9c24561ef41ee91bb4a6a2f75bde907e99a399ddef66a1825b02eb6e4e478879"}
Nov 01 21:08:09.636168 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.635807    4872 scope.go:122] "RemoveContainer" containerID="78a64526cae75720136b6463645a78b07864d68b196cace04395b27d077f4ad9"
Nov 01 21:08:09.796612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.796547    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 21:08:09.896276 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896244    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896475 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896359    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896475 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896458    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896585 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896492    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896585 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896497    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:09.896585 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896548    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896594    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896623    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896698    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896920 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896774    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket\") pod \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\" (UID: \"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7\") "
Nov 01 21:08:09.896920 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.896871    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:09.897041 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897003    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:09.897312 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897289    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:09.897415 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897310    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:08:09.897479 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897445    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.897479 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897467    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.897570 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897481    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.897570 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897495    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.897570 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.897509    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.898833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.898809    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:09.898833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.898819    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:08:09.899061 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.899048    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:08:09.899146 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.899128    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9" pod "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" (UID: "f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7"). InnerVolumeSpecName "kube-api-access-vnbg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:08:09.998447 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.998417    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbg9\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-kube-api-access-vnbg9\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.998447 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.998445    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.998669 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.998455    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:09.998669 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:09.998464    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:10.644460 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:10.644421    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"
Nov 01 21:08:10.644460 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:10.644441    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr" event={"ID":"f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7","Type":"ContainerDied","Data":"44d2c47f8897a12abef8643dea75a7996a1028ac199f66c567ad44b0711c1a97"}
Nov 01 21:08:10.645012 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:10.644507    4872 scope.go:122] "RemoveContainer" containerID="9c24561ef41ee91bb4a6a2f75bde907e99a399ddef66a1825b02eb6e4e478879"
Nov 01 21:08:10.866093 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:10.866054    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"]
Nov 01 21:08:10.915873 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:10.915837    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-sz2sr"]
Nov 01 21:08:11.079367 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:11.079332    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" path="/var/lib/kubelet/pods/f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7/volumes"
Nov 01 21:08:23.366902 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:08:23.366854    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:08:32.165400 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.165360    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"]
Nov 01 21:08:32.165978 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.165961    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy"
Nov 01 21:08:32.165978 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.165977    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy"
Nov 01 21:08:32.166099 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.165988    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy"
Nov 01 21:08:32.166099 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.165993    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy"
Nov 01 21:08:32.166099 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.166072    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy"
Nov 01 21:08:32.166099 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.166081    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eb4e5f-4d3d-4205-b4ca-29a17e6965b7" containerName="istio-proxy"
Nov 01 21:08:32.171050 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.171029    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.227209 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.227182    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"]
Nov 01 21:08:32.328777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.328746    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.328954 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.328787    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.328954 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.328804    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.329047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.328964    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.329047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.329002    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.329047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.329018    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5ld\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.329047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.329039    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.329243 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.329221    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.329296 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.329255    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430721 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430632    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430721 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430683    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430747    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430771    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430793    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430853    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430882    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430899    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5ld\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.430927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.430916    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.431304 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.431042    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.431304 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.431073    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.431446 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.431424    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.431519 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.431452    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.431866 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.431841    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.433133 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.433114    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.433191 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.433158    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.488442 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.488413    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.488548 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.488415    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5ld\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld\") pod \"maas-default-gateway-openshift-default-58c6585447-qps6t\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:32.785309 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:32.785210    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:33.220048 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.220015    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"]
Nov 01 21:08:33.261142 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:08:33.261104    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2637e83c_ebe9_41ba_89b1_10d57a44ff6d.slice/crio-0591c3e292afea6b6fbc018193788a1d21717419f7627e692b899d275371181c WatchSource:0}: Error finding container 0591c3e292afea6b6fbc018193788a1d21717419f7627e692b899d275371181c: Status 404 returned error can't find the container with id 0591c3e292afea6b6fbc018193788a1d21717419f7627e692b899d275371181c
Nov 01 21:08:33.263897 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.263864    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:08:33.263984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.263932    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:08:33.263984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.263962    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:08:33.810638 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.810597    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" event={"ID":"2637e83c-ebe9-41ba-89b1-10d57a44ff6d","Type":"ContainerStarted","Data":"0d255a9ba24bc2c5ee63727c465cca95710ef1138d75be06b4f3fbd37c34adf9"}
Nov 01 21:08:33.810846 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.810670    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" event={"ID":"2637e83c-ebe9-41ba-89b1-10d57a44ff6d","Type":"ContainerStarted","Data":"0591c3e292afea6b6fbc018193788a1d21717419f7627e692b899d275371181c"}
Nov 01 21:08:33.948627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:33.948568    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podStartSLOduration=1.948549033 podStartE2EDuration="1.948549033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:33.946753478 +0000 UTC m=+2270.995863298" watchObservedRunningTime="2026-11-01 21:08:33.948549033 +0000 UTC m=+2270.997658858"
Nov 01 21:08:34.785932 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:34.785897    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:34.788311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:34.787219    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:34.788311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:34.787272    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:35.785938 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:35.785902    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:35.786360 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:35.785956    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:36.786130 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:36.786095    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:36.786620 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:36.786153    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:37.785738 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:37.785701    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:37.785937 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:37.785760    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:38.786789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:38.786750    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:38.787195 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:38.786824    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:39.786129 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:39.786089    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:39.786341 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:39.786155    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:40.785938 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:40.785902    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qps6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 21:08:40.786493 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:40.785970    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 21:08:41.790496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:41.790468    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:41.791009 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:41.790899    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:41.791574 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:08:41.791555    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:08:53.373164 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:08:53.373125    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:09:19.060601 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.060569    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"]
Nov 01 21:09:19.111191 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.111163    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"]
Nov 01 21:09:19.113813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.113783    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"]
Nov 01 21:09:19.897936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.897902    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"]
Nov 01 21:09:19.912218 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.910852    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:19.954107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.954067    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"]
Nov 01 21:09:19.969446 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.969413    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-670df9\""
Nov 01 21:09:19.969757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.969585    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-670df9-dockercfg-k5kx9\""
Nov 01 21:09:19.989108 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.989059    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") " pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:19.989334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:19.989302    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh28m\" (UniqueName: \"kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m\") pod \"payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") " pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:20.090673 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:20.090629    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh28m\" (UniqueName: \"kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m\") pod \"payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") " pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:20.091086 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:20.090823    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") " pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:20.091753 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:20.091728    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") " pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:20.156743 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:20.156668    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh28m\" (UniqueName: \"kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m\") pod \"payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") " pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:20.234227 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:20.234197    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:20.616508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:20.616471    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"]
Nov 01 21:09:20.618251 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:09:20.618223    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6988e2ad_3e8b_433c_9491_3035033a73da.slice/crio-268104471a7c357945a044ba5b0325a02eb8f80525d752cda969cdd74c3632d6 WatchSource:0}: Error finding container 268104471a7c357945a044ba5b0325a02eb8f80525d752cda969cdd74c3632d6: Status 404 returned error can't find the container with id 268104471a7c357945a044ba5b0325a02eb8f80525d752cda969cdd74c3632d6
Nov 01 21:09:21.113119 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:21.113087    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck" event={"ID":"6988e2ad-3e8b-433c-9491-3035033a73da","Type":"ContainerStarted","Data":"39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966"}
Nov 01 21:09:21.113528 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:21.113133    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck" event={"ID":"6988e2ad-3e8b-433c-9491-3035033a73da","Type":"ContainerStarted","Data":"268104471a7c357945a044ba5b0325a02eb8f80525d752cda969cdd74c3632d6"}
Nov 01 21:09:21.113528 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:21.113165    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:21.229515 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:21.229466    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck" podStartSLOduration=2.229452777 podStartE2EDuration="2.229452777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:21.2273404 +0000 UTC m=+2318.276450220" watchObservedRunningTime="2026-11-01 21:09:21.229452777 +0000 UTC m=+2318.278562597"
Nov 01 21:09:23.365797 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:23.365760    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:09:28.060104 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.059707    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"]
Nov 01 21:09:28.079254 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.079213    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.113909 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.113868    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"]
Nov 01 21:09:28.151095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.151064    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-044440\""
Nov 01 21:09:28.151095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.151076    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-044440-dockercfg-kfmzl\""
Nov 01 21:09:28.202640 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.202609    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-044440-5678b676c-cg7nt\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") " pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.202830 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.202821    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqs9p\" (UniqueName: \"kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p\") pod \"payload-processing-e2e-inf-b-044440-5678b676c-cg7nt\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") " pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.304506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.304476    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-044440-5678b676c-cg7nt\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") " pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.304703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.304526    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqs9p\" (UniqueName: \"kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p\") pod \"payload-processing-e2e-inf-b-044440-5678b676c-cg7nt\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") " pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.305315 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.305291    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-044440-5678b676c-cg7nt\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") " pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.362544 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.362480    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqs9p\" (UniqueName: \"kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p\") pod \"payload-processing-e2e-inf-b-044440-5678b676c-cg7nt\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") " pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.408953 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.408924    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:28.783924 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:28.783885    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"]
Nov 01 21:09:28.785941 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:09:28.785905    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e39be9c_5579_4fe3_afa0_c99472f7d497.slice/crio-13264f2b3a8667d1a9b8d5c53e791c9102984e3dc2d08ad790c449ff478aace8 WatchSource:0}: Error finding container 13264f2b3a8667d1a9b8d5c53e791c9102984e3dc2d08ad790c449ff478aace8: Status 404 returned error can't find the container with id 13264f2b3a8667d1a9b8d5c53e791c9102984e3dc2d08ad790c449ff478aace8
Nov 01 21:09:29.167834 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:29.167795    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt" event={"ID":"0e39be9c-5579-4fe3-afa0-c99472f7d497","Type":"ContainerStarted","Data":"2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0"}
Nov 01 21:09:29.168325 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:29.167869    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt" event={"ID":"0e39be9c-5579-4fe3-afa0-c99472f7d497","Type":"ContainerStarted","Data":"13264f2b3a8667d1a9b8d5c53e791c9102984e3dc2d08ad790c449ff478aace8"}
Nov 01 21:09:29.168325 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:29.167975    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:29.283476 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:29.283416    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt" podStartSLOduration=2.28339786 podStartE2EDuration="2.28339786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:29.28198934 +0000 UTC m=+2326.331099159" watchObservedRunningTime="2026-11-01 21:09:29.28339786 +0000 UTC m=+2326.332507679"
Nov 01 21:09:32.122266 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:32.122234    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:09:34.081348 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.081306    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main" containerID="cri-o://11b22e144b66fdb31094e88e6e6b417ee860f4063d2d490220dd07fb259b5f68" gracePeriod=45
Nov 01 21:09:34.132521 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.132458    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main" containerID="cri-o://646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb" gracePeriod=45
Nov 01 21:09:34.134370 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.134336    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main" containerID="cri-o://e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c" gracePeriod=45
Nov 01 21:09:34.200292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.200261    4872 generic.go:356] "Generic (PLEG): container finished" podID="e76854fd-0531-48e5-9f07-5592acd1d893" containerID="11b22e144b66fdb31094e88e6e6b417ee860f4063d2d490220dd07fb259b5f68" exitCode=0
Nov 01 21:09:34.200414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.200303    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" event={"ID":"e76854fd-0531-48e5-9f07-5592acd1d893","Type":"ContainerDied","Data":"11b22e144b66fdb31094e88e6e6b417ee860f4063d2d490220dd07fb259b5f68"}
Nov 01 21:09:34.477839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.477797    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 21:09:34.481090 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.481070    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 21:09:34.484807 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.484788    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 21:09:34.537283 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537227    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache\") pod \"f611828c-fb69-406f-a4a0-6d1c85c49183\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") "
Nov 01 21:09:34.537467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537349    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir\") pod \"e76854fd-0531-48e5-9f07-5592acd1d893\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") "
Nov 01 21:09:34.537467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537414    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs\" (UniqueName: \"kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs\") pod \"f611828c-fb69-406f-a4a0-6d1c85c49183\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") "
Nov 01 21:09:34.537467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537462    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs\" (UniqueName: \"kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs\") pod \"e76854fd-0531-48e5-9f07-5592acd1d893\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") "
Nov 01 21:09:34.537644 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537529    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache" pod "f611828c-fb69-406f-a4a0-6d1c85c49183" (UID: "f611828c-fb69-406f-a4a0-6d1c85c49183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.537753 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537565    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home\") pod \"f611828c-fb69-406f-a4a0-6d1c85c49183\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") "
Nov 01 21:09:34.537887 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537778    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home" pod "f611828c-fb69-406f-a4a0-6d1c85c49183" (UID: "f611828c-fb69-406f-a4a0-6d1c85c49183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.537887 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537819    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm\") pod \"f611828c-fb69-406f-a4a0-6d1c85c49183\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") "
Nov 01 21:09:34.537887 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537868    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home\") pod \"e76854fd-0531-48e5-9f07-5592acd1d893\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") "
Nov 01 21:09:34.538053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537928    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm\") pod \"e76854fd-0531-48e5-9f07-5592acd1d893\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") "
Nov 01 21:09:34.538053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537956    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir" pod "e76854fd-0531-48e5-9f07-5592acd1d893" (UID: "e76854fd-0531-48e5-9f07-5592acd1d893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.538053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.537968    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq\" (UniqueName: \"kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq\") pod \"e76854fd-0531-48e5-9f07-5592acd1d893\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") "
Nov 01 21:09:34.538163 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.538115    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home" pod "e76854fd-0531-48e5-9f07-5592acd1d893" (UID: "e76854fd-0531-48e5-9f07-5592acd1d893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.538342    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs\" (UniqueName: \"kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs\") pod \"f611828c-fb69-406f-a4a0-6d1c85c49183\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") "
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.538391    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir\") pod \"f611828c-fb69-406f-a4a0-6d1c85c49183\" (UID: \"f611828c-fb69-406f-a4a0-6d1c85c49183\") "
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.538493    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache\") pod \"e76854fd-0531-48e5-9f07-5592acd1d893\" (UID: \"e76854fd-0531-48e5-9f07-5592acd1d893\") "
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.538797    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir" pod "f611828c-fb69-406f-a4a0-6d1c85c49183" (UID: "f611828c-fb69-406f-a4a0-6d1c85c49183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.538954    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache" pod "e76854fd-0531-48e5-9f07-5592acd1d893" (UID: "e76854fd-0531-48e5-9f07-5592acd1d893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.539175    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.539187    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.539197    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.539204    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.539211    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.539219    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.540130    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs" pod "f611828c-fb69-406f-a4a0-6d1c85c49183" (UID: "f611828c-fb69-406f-a4a0-6d1c85c49183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.540279    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm" pod "f611828c-fb69-406f-a4a0-6d1c85c49183" (UID: "f611828c-fb69-406f-a4a0-6d1c85c49183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.540388    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq" pod "e76854fd-0531-48e5-9f07-5592acd1d893" (UID: "e76854fd-0531-48e5-9f07-5592acd1d893"). InnerVolumeSpecName "kube-api-access-pfxvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:09:34.540700 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.540581    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs" pod "f611828c-fb69-406f-a4a0-6d1c85c49183" (UID: "f611828c-fb69-406f-a4a0-6d1c85c49183"). InnerVolumeSpecName "kube-api-access-q7hcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:09:34.541624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.541460    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm" pod "e76854fd-0531-48e5-9f07-5592acd1d893" (UID: "e76854fd-0531-48e5-9f07-5592acd1d893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.541624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.541521    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs" pod "e76854fd-0531-48e5-9f07-5592acd1d893" (UID: "e76854fd-0531-48e5-9f07-5592acd1d893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.639784    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home\") pod \"db47e85a-b794-4b0e-b415-83a7191b4672\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") "
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.639833    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache\") pod \"db47e85a-b794-4b0e-b415-83a7191b4672\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") "
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.639861    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm\") pod \"db47e85a-b794-4b0e-b415-83a7191b4672\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") "
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.639920    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs\" (UniqueName: \"kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs\") pod \"db47e85a-b794-4b0e-b415-83a7191b4672\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") "
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.639956    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw\" (UniqueName: \"kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw\") pod \"db47e85a-b794-4b0e-b415-83a7191b4672\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") "
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640160    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home" pod "db47e85a-b794-4b0e-b415-83a7191b4672" (UID: "db47e85a-b794-4b0e-b415-83a7191b4672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640187    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir\") pod \"db47e85a-b794-4b0e-b415-83a7191b4672\" (UID: \"db47e85a-b794-4b0e-b415-83a7191b4672\") "
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640584    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache" pod "db47e85a-b794-4b0e-b415-83a7191b4672" (UID: "db47e85a-b794-4b0e-b415-83a7191b4672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640905    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f611828c-fb69-406f-a4a0-6d1c85c49183-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640931    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76854fd-0531-48e5-9f07-5592acd1d893-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640944    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f611828c-fb69-406f-a4a0-6d1c85c49183-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640957    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76854fd-0531-48e5-9f07-5592acd1d893-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640972    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxvq\" (UniqueName: \"kubernetes.io/projected/e76854fd-0531-48e5-9f07-5592acd1d893-kube-api-access-pfxvq\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.640987    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hcs\" (UniqueName: \"kubernetes.io/projected/f611828c-fb69-406f-a4a0-6d1c85c49183-kube-api-access-q7hcs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.641001    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.641014    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.641886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.641358    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir" pod "db47e85a-b794-4b0e-b415-83a7191b4672" (UID: "db47e85a-b794-4b0e-b415-83a7191b4672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.643020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.642989    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm" pod "db47e85a-b794-4b0e-b415-83a7191b4672" (UID: "db47e85a-b794-4b0e-b415-83a7191b4672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:09:34.643020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.643006    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs" pod "db47e85a-b794-4b0e-b415-83a7191b4672" (UID: "db47e85a-b794-4b0e-b415-83a7191b4672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:09:34.643479 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.643454    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw" pod "db47e85a-b794-4b0e-b415-83a7191b4672" (UID: "db47e85a-b794-4b0e-b415-83a7191b4672"). InnerVolumeSpecName "kube-api-access-dthrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:09:34.742345 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.742308    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.742345 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.742339    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db47e85a-b794-4b0e-b415-83a7191b4672-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.742345 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.742348    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db47e85a-b794-4b0e-b415-83a7191b4672-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:34.742637 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:34.742357    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dthrw\" (UniqueName: \"kubernetes.io/projected/db47e85a-b794-4b0e-b415-83a7191b4672-kube-api-access-dthrw\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:35.214676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.213230    4872 generic.go:356] "Generic (PLEG): container finished" podID="db47e85a-b794-4b0e-b415-83a7191b4672" containerID="e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c" exitCode=0
Nov 01 21:09:35.214676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.213447    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" event={"ID":"db47e85a-b794-4b0e-b415-83a7191b4672","Type":"ContainerDied","Data":"e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c"}
Nov 01 21:09:35.214676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.213518    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n" event={"ID":"db47e85a-b794-4b0e-b415-83a7191b4672","Type":"ContainerDied","Data":"5c7f5385203c70b34994daf6739adebe2888e082420509f35cacf85b0fb392d0"}
Nov 01 21:09:35.214676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.213570    4872 scope.go:122] "RemoveContainer" containerID="e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c"
Nov 01 21:09:35.214676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.213357    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"
Nov 01 21:09:35.219051 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.216300    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g" event={"ID":"e76854fd-0531-48e5-9f07-5592acd1d893","Type":"ContainerDied","Data":"5c53ccf68d5dd251b175459cdbd779531884908bc1b02c4374e64f86aef2c680"}
Nov 01 21:09:35.219051 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.216386    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"
Nov 01 21:09:35.221711 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.219539    4872 generic.go:356] "Generic (PLEG): container finished" podID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerID="646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb" exitCode=0
Nov 01 21:09:35.221711 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.219611    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" event={"ID":"f611828c-fb69-406f-a4a0-6d1c85c49183","Type":"ContainerDied","Data":"646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb"}
Nov 01 21:09:35.221711 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.219639    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph" event={"ID":"f611828c-fb69-406f-a4a0-6d1c85c49183","Type":"ContainerDied","Data":"0ead97893e60b106a4dfa2f346495c1df837db581ea488e8dac8f508f225820f"}
Nov 01 21:09:35.221711 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.219734    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"
Nov 01 21:09:35.236597 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.236575    4872 scope.go:122] "RemoveContainer" containerID="e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c"
Nov 01 21:09:35.236984 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:35.236957    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c\": container with ID starting with e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c not found: ID does not exist" containerID="e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c"
Nov 01 21:09:35.237079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.236995    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c"} err="failed to get container status \"e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c\": rpc error: code = NotFound desc = could not find container \"e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c\": container with ID starting with e2b4e9d01e99f8cd4ffc5232f9b24a87f979432aafa222d2a92d638dbfa0f06c not found: ID does not exist"
Nov 01 21:09:35.237079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.237019    4872 scope.go:122] "RemoveContainer" containerID="11b22e144b66fdb31094e88e6e6b417ee860f4063d2d490220dd07fb259b5f68"
Nov 01 21:09:35.254857 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.254137    4872 scope.go:122] "RemoveContainer" containerID="646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb"
Nov 01 21:09:35.266444 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.266424    4872 scope.go:122] "RemoveContainer" containerID="646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb"
Nov 01 21:09:35.266719 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:35.266700    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb\": container with ID starting with 646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb not found: ID does not exist" containerID="646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb"
Nov 01 21:09:35.266796 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.266725    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb"} err="failed to get container status \"646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb\": rpc error: code = NotFound desc = could not find container \"646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb\": container with ID starting with 646701cdf4987d5640e104df47eeecc070cff9886cd8c1a7d83a9a2ab58cd3cb not found: ID does not exist"
Nov 01 21:09:35.383572 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.383482    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"]
Nov 01 21:09:35.448679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.439946    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/premium-w6-dd038b-kserve-59547547b5-8xpph"]
Nov 01 21:09:35.561587 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.561552    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"]
Nov 01 21:09:35.564069 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.564043    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-w6-dd038b-kserve-75b476dcb8-pn92g"]
Nov 01 21:09:35.673235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.673195    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"]
Nov 01 21:09:35.677581 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:35.677558    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/simulator-w6-dd038b-kserve-6f89d55d48-t755n"]
Nov 01 21:09:36.565891 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:36.565773    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-nd97z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:09:36.565891 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:36.565841    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-nd97z" podUID="4102344f-e1f1-4c5a-a1b9-a57ba4e86b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:09:37.079719 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:37.079688    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" path="/var/lib/kubelet/pods/db47e85a-b794-4b0e-b415-83a7191b4672/volumes"
Nov 01 21:09:37.080170 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:37.080157    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" path="/var/lib/kubelet/pods/e76854fd-0531-48e5-9f07-5592acd1d893/volumes"
Nov 01 21:09:37.080558 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:37.080545    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" path="/var/lib/kubelet/pods/f611828c-fb69-406f-a4a0-6d1c85c49183/volumes"
Nov 01 21:09:40.177981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:40.177945    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:09:49.117044 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117014    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"]
Nov 01 21:09:49.117821 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117806    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main"
Nov 01 21:09:49.117885 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117824    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main"
Nov 01 21:09:49.117885 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117864    4872 cpu_manager.go:414] "containerMap: removing container" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main"
Nov 01 21:09:49.117885 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117871    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main"
Nov 01 21:09:49.117981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117907    4872 cpu_manager.go:414] "containerMap: removing container" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main"
Nov 01 21:09:49.117981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.117914    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main"
Nov 01 21:09:49.118046 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.118036    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="db47e85a-b794-4b0e-b415-83a7191b4672" containerName="main"
Nov 01 21:09:49.118075 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.118052    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76854fd-0531-48e5-9f07-5592acd1d893" containerName="main"
Nov 01 21:09:49.118075 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.118065    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f611828c-fb69-406f-a4a0-6d1c85c49183" containerName="main"
Nov 01 21:09:49.123539 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.123514    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.179326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.179294    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"]
Nov 01 21:09:49.226174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.226142    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3ab520-serving-cert\""
Nov 01 21:09:49.229635 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.229604    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.229776 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.229647    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.229858 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.229826    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw27p\" (UniqueName: \"kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.229858 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.229854    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.331292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.331256    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.331465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.331310    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.331465 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:49.331397    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:49.331465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.331413    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw27p\" (UniqueName: \"kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.331465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.331441    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.331465 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:49.331462    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:09:49.831443 +0000 UTC m=+2346.880552810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:49.332068 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.332050    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.333757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.333739    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.390831 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.390757    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw27p\" (UniqueName: \"kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.837935 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:49.837905    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:49.838104 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:49.838045    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:49.838145 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:49.838113    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:09:50.838099791 +0000 UTC m=+2347.887209604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:50.851380 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:50.851347    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:50.851882 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:50.851492    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:50.851882 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:50.851565    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:09:52.851546126 +0000 UTC m=+2349.900655939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:52.879457 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:52.879421    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:52.879864 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:52.879576    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:52.879864 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:52.879638    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:09:56.879623579 +0000 UTC m=+2353.928733383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:53.366491 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:53.366439    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:09:56.927771 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:09:56.927737    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:09:56.928188 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:56.927872    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:09:56.928188 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:09:56.927936    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:10:04.927921564 +0000 UTC m=+2361.977031373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:10:04.933333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:04.933297    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:10:04.933820 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:04.933425    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:10:04.933820 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:04.933489    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:10:20.933475834 +0000 UTC m=+2377.982585644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:10:12.999607 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:12.999572    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"]
Nov 01 21:10:13.003738 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.003720    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.060032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.060002    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"]
Nov 01 21:10:13.107263 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.107233    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7091fd-serving-cert\""
Nov 01 21:10:13.139250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.139217    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp5t\" (UniqueName: \"kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.139437 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.139355    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.139437 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.139424    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.139559 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.139520    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.241205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.241163    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp5t\" (UniqueName: \"kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.241648 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.241557    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.241831 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.241809    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.242025 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.242008    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.242376 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:13.242353    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-serving-cert: secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:13.242561 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:13.242550    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls podName:aa0796ee-20f5-40d4-ac70-98ca2694840c nodeName:}" failed. No retries permitted until 2026-11-01 21:10:13.742531005 +0000 UTC m=+2370.791640806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls") pod "maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c") : secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:13.242768 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.242741    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.253782 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.253697    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.302237 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.302203    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp5t\" (UniqueName: \"kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.749461 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:13.749426    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:13.749648 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:13.749573    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-serving-cert: secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:13.749730 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:13.749687    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls podName:aa0796ee-20f5-40d4-ac70-98ca2694840c nodeName:}" failed. No retries permitted until 2026-11-01 21:10:14.749648325 +0000 UTC m=+2371.798758134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls") pod "maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c") : secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:14.760354 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:14.760317    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:14.760792 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:14.760464    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-serving-cert: secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:14.760792 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:14.760531    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls podName:aa0796ee-20f5-40d4-ac70-98ca2694840c nodeName:}" failed. No retries permitted until 2026-11-01 21:10:16.760517943 +0000 UTC m=+2373.809627740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls") pod "maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c") : secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:16.785011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:16.784977    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:16.785396 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:16.785139    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-serving-cert: secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:16.785396 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:16.785205    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls podName:aa0796ee-20f5-40d4-ac70-98ca2694840c nodeName:}" failed. No retries permitted until 2026-11-01 21:10:20.78519005 +0000 UTC m=+2377.834299862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls") pod "maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c") : secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:20.836037 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:20.835998    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:20.836465 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:20.836140    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-serving-cert: secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:20.836465 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:20.836208    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls podName:aa0796ee-20f5-40d4-ac70-98ca2694840c nodeName:}" failed. No retries permitted until 2026-11-01 21:10:28.836193952 +0000 UTC m=+2385.885303749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls") pod "maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c") : secret "maas-api-e2e-inf-b-7091fd-serving-cert" not found
Nov 01 21:10:20.937472 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:20.937429    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:10:20.937727 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:20.937607    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-serving-cert: secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:10:20.937727 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:20.937725    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls podName:f2d7d2ba-ebef-4765-afb5-e55a6e146e86 nodeName:}" failed. No retries permitted until 2026-11-01 21:10:52.937705539 +0000 UTC m=+2409.986815335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls") pod "maas-api-e2e-inf-a-3ab520-744969b995-sslqx" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86") : secret "maas-api-e2e-inf-a-3ab520-serving-cert" not found
Nov 01 21:10:23.366608 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:23.366564    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:10:28.841784 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:28.841690    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:28.844420 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:28.844395    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:28.919129 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:28.919094    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:29.301628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:29.301602    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"]
Nov 01 21:10:29.303507 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:10:29.303477    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0796ee_20f5_40d4_ac70_98ca2694840c.slice/crio-787f1855b942ca6341c1115571a1281033a616145feb695d638859196815abd5 WatchSource:0}: Error finding container 787f1855b942ca6341c1115571a1281033a616145feb695d638859196815abd5: Status 404 returned error can't find the container with id 787f1855b942ca6341c1115571a1281033a616145feb695d638859196815abd5
Nov 01 21:10:29.595702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:29.595590    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" event={"ID":"aa0796ee-20f5-40d4-ac70-98ca2694840c","Type":"ContainerStarted","Data":"787f1855b942ca6341c1115571a1281033a616145feb695d638859196815abd5"}
Nov 01 21:10:30.603003 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:30.602964    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" event={"ID":"aa0796ee-20f5-40d4-ac70-98ca2694840c","Type":"ContainerStarted","Data":"74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0"}
Nov 01 21:10:30.603429 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:30.603040    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:30.721679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:30.721607    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" podStartSLOduration=17.922773253 podStartE2EDuration="18.721586655s" totalImagesPullingTime="798.813402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:10:30.718202874 +0000 UTC m=+2387.767312693" watchObservedRunningTime="2026-11-01 21:10:30.721586655 +0000 UTC m=+2387.770696473"
Nov 01 21:10:36.615155 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:36.615123    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:10:43.535847 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:43.535818    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:10:43.536289 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:43.536246    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:10:43.541712 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:43.541687    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:10:43.542153 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:43.542134    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:10:51.749896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.749861    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"]
Nov 01 21:10:51.754108 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.754083    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.809720 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.809687    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"]
Nov 01 21:10:51.856007 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.855971    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-044440-serving-cert\""
Nov 01 21:10:51.876369 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.876333    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.876541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.876502    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.876587 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.876567    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.876691 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.876674    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnl4\" (UniqueName: \"kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.977516 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.977476    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.977777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.977609    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.977777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.977721    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.977905 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.977775    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnl4\" (UniqueName: \"kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.977905 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:51.977854    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-serving-cert: secret "maas-api-e2e-inf-b-044440-serving-cert" not found
Nov 01 21:10:51.977982 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:51.977946    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls podName:deccf8ce-cbdd-41f1-b1a8-f105b0394e7b nodeName:}" failed. No retries permitted until 2026-11-01 21:10:52.477925528 +0000 UTC m=+2409.527035327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls") pod "maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" (UID: "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b") : secret "maas-api-e2e-inf-b-044440-serving-cert" not found
Nov 01 21:10:51.978242 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.978223    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:51.980179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:51.980160    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:52.045692 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:52.045586    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnl4\" (UniqueName: \"kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:52.487083 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:52.487049    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:52.489502 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:52.489473    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-044440-65d44d9979-xh7bf\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:52.669273 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:52.669239    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:52.995508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:52.995477    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:10:52.997937 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:52.997912    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ab520-744969b995-sslqx\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:10:53.037643 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:53.037609    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:10:53.045840 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:53.045819    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"]
Nov 01 21:10:53.048971 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:10:53.048935    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeccf8ce_cbdd_41f1_b1a8_f105b0394e7b.slice/crio-ef6b88afcf5c008705628388335b9ed710d59d83c4e566fcca3c8863eba2bb89 WatchSource:0}: Error finding container ef6b88afcf5c008705628388335b9ed710d59d83c4e566fcca3c8863eba2bb89: Status 404 returned error can't find the container with id ef6b88afcf5c008705628388335b9ed710d59d83c4e566fcca3c8863eba2bb89
Nov 01 21:10:53.367813 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:10:53.367724    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:10:53.428717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:53.428684    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"]
Nov 01 21:10:53.430069 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:10:53.430040    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d7d2ba_ebef_4765_afb5_e55a6e146e86.slice/crio-a3797963dbac30554df8b804fcdd68e0eff261e3548952861ee610eea955294d WatchSource:0}: Error finding container a3797963dbac30554df8b804fcdd68e0eff261e3548952861ee610eea955294d: Status 404 returned error can't find the container with id a3797963dbac30554df8b804fcdd68e0eff261e3548952861ee610eea955294d
Nov 01 21:10:53.775517 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:53.772694    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" event={"ID":"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b","Type":"ContainerStarted","Data":"ef6b88afcf5c008705628388335b9ed710d59d83c4e566fcca3c8863eba2bb89"}
Nov 01 21:10:53.775517 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:53.774632    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx" event={"ID":"f2d7d2ba-ebef-4765-afb5-e55a6e146e86","Type":"ContainerStarted","Data":"a3797963dbac30554df8b804fcdd68e0eff261e3548952861ee610eea955294d"}
Nov 01 21:10:54.783264 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:54.783231    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" event={"ID":"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b","Type":"ContainerStarted","Data":"529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17"}
Nov 01 21:10:54.783814 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:54.783422    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:10:54.784959 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:54.784932    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx" event={"ID":"f2d7d2ba-ebef-4765-afb5-e55a6e146e86","Type":"ContainerStarted","Data":"8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8"}
Nov 01 21:10:54.785086 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:54.785070    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:10:54.901692 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:54.901606    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" podStartSLOduration=3.128233745 podStartE2EDuration="3.901591092s" totalImagesPullingTime="773.357347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:10:54.89888051 +0000 UTC m=+2411.947990324" watchObservedRunningTime="2026-11-01 21:10:54.901591092 +0000 UTC m=+2411.950700911"
Nov 01 21:10:55.028495 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:10:55.027976    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx" podStartSLOduration=65.133742098 podStartE2EDuration="1m6.027952548s" totalImagesPullingTime="894.21045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:10:55.015370392 +0000 UTC m=+2412.064480210" watchObservedRunningTime="2026-11-01 21:10:55.027952548 +0000 UTC m=+2412.077062369"
Nov 01 21:11:00.795584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:00.795540    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:11:00.796061 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:00.796000    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:11:23.374236 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:11:23.374182    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:11:53.369904 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:11:53.369866    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:11:57.099789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.099697    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"]
Nov 01 21:11:57.100295 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.099916    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" podUID="e9380fa8-60b4-49f5-9653-a6940faa6b9e" containerName="maas-api" containerID="cri-o://a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec" gracePeriod=30
Nov 01 21:11:57.400570 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.400542    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 21:11:57.480023 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.479987    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs\") pod \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") "
Nov 01 21:11:57.480206 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.480056    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca\") pod \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") "
Nov 01 21:11:57.480206 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.480137    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\") pod \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") "
Nov 01 21:11:57.480206 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.480178    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr\" (UniqueName: \"kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr\") pod \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\" (UID: \"e9380fa8-60b4-49f5-9653-a6940faa6b9e\") "
Nov 01 21:11:57.480454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.480430    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca" pod "e9380fa8-60b4-49f5-9653-a6940faa6b9e" (UID: "e9380fa8-60b4-49f5-9653-a6940faa6b9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:11:57.480829 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.480808    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9380fa8-60b4-49f5-9653-a6940faa6b9e-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:57.482384 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.482365    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs" pod "e9380fa8-60b4-49f5-9653-a6940faa6b9e" (UID: "e9380fa8-60b4-49f5-9653-a6940faa6b9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:11:57.482485 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.482396    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls" pod "e9380fa8-60b4-49f5-9653-a6940faa6b9e" (UID: "e9380fa8-60b4-49f5-9653-a6940faa6b9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:11:57.482595 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.482576    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr" pod "e9380fa8-60b4-49f5-9653-a6940faa6b9e" (UID: "e9380fa8-60b4-49f5-9653-a6940faa6b9e"). InnerVolumeSpecName "kube-api-access-t5fnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:11:57.582153 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.582115    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:57.582153 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.582147    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9380fa8-60b4-49f5-9653-a6940faa6b9e-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:57.582153 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:57.582157    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fnr\" (UniqueName: \"kubernetes.io/projected/e9380fa8-60b4-49f5-9653-a6940faa6b9e-kube-api-access-t5fnr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:58.207205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.207170    4872 generic.go:356] "Generic (PLEG): container finished" podID="e9380fa8-60b4-49f5-9653-a6940faa6b9e" containerID="a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec" exitCode=0
Nov 01 21:11:58.208994 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.208965    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" event={"ID":"e9380fa8-60b4-49f5-9653-a6940faa6b9e","Type":"ContainerDied","Data":"a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec"}
Nov 01 21:11:58.209218 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.209196    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986" event={"ID":"e9380fa8-60b4-49f5-9653-a6940faa6b9e","Type":"ContainerDied","Data":"2ae54938b6a121460b933af0fa5f623088a0827d4cb8b477e7e7b4118f5cf0b5"}
Nov 01 21:11:58.209389 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.209376    4872 scope.go:122] "RemoveContainer" containerID="a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec"
Nov 01 21:11:58.209624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.209610    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"
Nov 01 21:11:58.238831 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.238805    4872 scope.go:122] "RemoveContainer" containerID="a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec"
Nov 01 21:11:58.239201 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:11:58.239174    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec\": container with ID starting with a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec not found: ID does not exist" containerID="a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec"
Nov 01 21:11:58.239445 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.239210    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec"} err="failed to get container status \"a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec\": rpc error: code = NotFound desc = could not find container \"a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec\": container with ID starting with a6dd1f0d54fc99eec300d53f151b18553d7b3477aaa7b26944100ae2a292a4ec not found: ID does not exist"
Nov 01 21:11:58.384723 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.384617    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"]
Nov 01 21:11:58.437149 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:58.437105    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd038b-85f5d869bd-5c986"]
Nov 01 21:11:59.080541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.080506    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9380fa8-60b4-49f5-9653-a6940faa6b9e" path="/var/lib/kubelet/pods/e9380fa8-60b4-49f5-9653-a6940faa6b9e/volumes"
Nov 01 21:11:59.610131 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.610095    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"]
Nov 01 21:11:59.610737 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.610720    4872 cpu_manager.go:414] "containerMap: removing container" podUID="e9380fa8-60b4-49f5-9653-a6940faa6b9e" containerName="maas-api"
Nov 01 21:11:59.610794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.610739    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9380fa8-60b4-49f5-9653-a6940faa6b9e" containerName="maas-api"
Nov 01 21:11:59.610871 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.610862    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9380fa8-60b4-49f5-9653-a6940faa6b9e" containerName="maas-api"
Nov 01 21:11:59.615664 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.615633    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.709706 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.709646    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxkf\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.709910 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.709797    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.709910 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.709856    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.709998 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.709980    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.710171 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.710145    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.710299 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.710280    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.710346 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.710331    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.710414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.710400    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.710454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.710428    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.770768 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.770733    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7091fd-openshift-default-dockercfg-pms7j\""
Nov 01 21:11:59.811486 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.811449    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.811703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.811559    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.811703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.811598    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.811703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.811628    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.811986 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.811959    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.812169 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.812151    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.812646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.812387    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.812757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.812541    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.812841 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.812820    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxkf\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.812994 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.812969    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.813174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.813024    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.813174 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.813102    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.814186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.814160    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.814326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.814252    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.814564 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.814545    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"]
Nov 01 21:11:59.815508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.815486    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.816480 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.816447    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.871411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.871339    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.871411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.871348    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxkf\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:11:59.929632 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:11:59.929596    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:12:00.364170 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:00.364142    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"]
Nov 01 21:12:00.365337 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:12:00.365310    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11a3ebc_7f1a_4273_a735_821126c88794.slice/crio-2e72692ba688e06d9e4ecc219422ff24c93fb286a74551d6a0384fee955fd372 WatchSource:0}: Error finding container 2e72692ba688e06d9e4ecc219422ff24c93fb286a74551d6a0384fee955fd372: Status 404 returned error can't find the container with id 2e72692ba688e06d9e4ecc219422ff24c93fb286a74551d6a0384fee955fd372
Nov 01 21:12:00.367936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:00.367905    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:12:00.368020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:00.367980    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:12:00.368020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:00.368009    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:12:01.232860 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:01.232817    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" event={"ID":"b11a3ebc-7f1a-4273-a735-821126c88794","Type":"ContainerStarted","Data":"e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e"}
Nov 01 21:12:01.233303 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:01.232883    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" event={"ID":"b11a3ebc-7f1a-4273-a735-821126c88794","Type":"ContainerStarted","Data":"2e72692ba688e06d9e4ecc219422ff24c93fb286a74551d6a0384fee955fd372"}
Nov 01 21:12:01.352874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:01.352819    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podStartSLOduration=2.352803884 podStartE2EDuration="2.352803884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:12:01.350543833 +0000 UTC m=+2478.399653654" watchObservedRunningTime="2026-11-01 21:12:01.352803884 +0000 UTC m=+2478.401913702"
Nov 01 21:12:01.935090 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:01.932007    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:12:01.940682 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:01.939213    4872 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 21:12:01.940682 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:01.939270    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 21:12:02.180674 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:02.179448    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"]
Nov 01 21:12:02.180674 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:02.179705    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" containerID="cri-o://fec81527bba23dae758484a974e222193475e6c6e6d294e05ea7b3a862090c62" gracePeriod=30
Nov 01 21:12:02.930235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:02.930203    4872 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 21:12:02.930674 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:02.930256    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 21:12:03.333341 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:03.333252    4872 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:12:03.333341 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:03.333314    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:12:03.930378 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:03.930339    4872 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 21:12:03.930934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:03.930401    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 21:12:04.930873 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:04.930839    4872 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 21:12:04.931356 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:04.930897    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 21:12:05.930006 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:05.929965    4872 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 21:12:05.930236 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:05.930029    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 21:12:06.935616 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:06.935588    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:12:06.936028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:06.935898    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:12:06.936752 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:06.936731    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:12:07.347907 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.347865    4872 generic.go:356] "Generic (PLEG): container finished" podID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerID="fec81527bba23dae758484a974e222193475e6c6e6d294e05ea7b3a862090c62" exitCode=0
Nov 01 21:12:07.348116 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.347928    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" event={"ID":"6d37b7fd-beb4-428c-8f6b-ca28097b15a9","Type":"ContainerDied","Data":"fec81527bba23dae758484a974e222193475e6c6e6d294e05ea7b3a862090c62"}
Nov 01 21:12:07.348116 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.347990    4872 scope.go:122] "RemoveContainer" containerID="1701d4ebc897a8b22bf1c64d0a54658f6c1726a27f78c96f9c334f17807b967b"
Nov 01 21:12:07.484590 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.484559    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 21:12:07.564843 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.564792    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.564861    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.564921    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.564981    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565058    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565100    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565128    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565163    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:12:07.565402 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565183    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565402 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565241    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:12:07.565402 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565278    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs\") pod \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\" (UID: \"6d37b7fd-beb4-428c-8f6b-ca28097b15a9\") "
Nov 01 21:12:07.565402 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565288    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:12:07.565402 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565366    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:12:07.565711 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565633    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:12:07.565844 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565829    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.565928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565847    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.565928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565861    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.565928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565875    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.565928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.565888    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.567385 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.567360    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:12:07.567507 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.567464    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:12:07.567778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.567759    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:12:07.567831 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.567805    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk" pod "6d37b7fd-beb4-428c-8f6b-ca28097b15a9" (UID: "6d37b7fd-beb4-428c-8f6b-ca28097b15a9"). InnerVolumeSpecName "kube-api-access-559qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:12:07.666940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.666909    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.666940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.666936    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.666940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.666947    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559qk\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-kube-api-access-559qk\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:07.667180 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:07.666957    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d37b7fd-beb4-428c-8f6b-ca28097b15a9-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:08.356177 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:08.356132    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l" event={"ID":"6d37b7fd-beb4-428c-8f6b-ca28097b15a9","Type":"ContainerDied","Data":"48099fbffa332b9cd0df625f9b7dc4411b7805fa4181037166727630af2e95e7"}
Nov 01 21:12:08.356619 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:08.356202    4872 scope.go:122] "RemoveContainer" containerID="fec81527bba23dae758484a974e222193475e6c6e6d294e05ea7b3a862090c62"
Nov 01 21:12:08.356619 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:08.356153    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"
Nov 01 21:12:08.524123 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:08.524089    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"]
Nov 01 21:12:08.574808 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:08.574774    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-h2t4l"]
Nov 01 21:12:09.079633 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:09.079603    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" path="/var/lib/kubelet/pods/6d37b7fd-beb4-428c-8f6b-ca28097b15a9/volumes"
Nov 01 21:12:23.365790 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:12:23.365753    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:12:52.956211 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:52.956174    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"]
Nov 01 21:12:52.956724 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:52.956423    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx" podUID="f2d7d2ba-ebef-4765-afb5-e55a6e146e86" containerName="maas-api" containerID="cri-o://8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8" gracePeriod=30
Nov 01 21:12:53.268673 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.268629    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:12:53.414451 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.412176    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") pod \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") "
Nov 01 21:12:53.414451 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.412230    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs\") pod \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") "
Nov 01 21:12:53.414451 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.412436    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca\") pod \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") "
Nov 01 21:12:53.414451 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.412579    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p\" (UniqueName: \"kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p\") pod \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\" (UID: \"f2d7d2ba-ebef-4765-afb5-e55a6e146e86\") "
Nov 01 21:12:53.418864 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:12:53.418826    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:12:53.419531 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.419299    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls" pod "f2d7d2ba-ebef-4765-afb5-e55a6e146e86" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:12:53.419732 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.419425    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca" pod "f2d7d2ba-ebef-4765-afb5-e55a6e146e86" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:12:53.428017 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.427970    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p" pod "f2d7d2ba-ebef-4765-afb5-e55a6e146e86" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86"). InnerVolumeSpecName "kube-api-access-vw27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:12:53.428238 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.428208    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs" pod "f2d7d2ba-ebef-4765-afb5-e55a6e146e86" (UID: "f2d7d2ba-ebef-4765-afb5-e55a6e146e86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:12:53.516746 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.515106    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw27p\" (UniqueName: \"kubernetes.io/projected/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-kube-api-access-vw27p\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:53.516746 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.515147    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:53.516746 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.515161    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:53.516746 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.515175    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d7d2ba-ebef-4765-afb5-e55a6e146e86-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:12:53.669170 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.669132    4872 generic.go:356] "Generic (PLEG): container finished" podID="f2d7d2ba-ebef-4765-afb5-e55a6e146e86" containerID="8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8" exitCode=0
Nov 01 21:12:53.669381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.669199    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"
Nov 01 21:12:53.669381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.669224    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx" event={"ID":"f2d7d2ba-ebef-4765-afb5-e55a6e146e86","Type":"ContainerDied","Data":"8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8"}
Nov 01 21:12:53.669381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.669297    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx" event={"ID":"f2d7d2ba-ebef-4765-afb5-e55a6e146e86","Type":"ContainerDied","Data":"a3797963dbac30554df8b804fcdd68e0eff261e3548952861ee610eea955294d"}
Nov 01 21:12:53.669381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.669332    4872 scope.go:122] "RemoveContainer" containerID="8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8"
Nov 01 21:12:53.684382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.684352    4872 scope.go:122] "RemoveContainer" containerID="8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8"
Nov 01 21:12:53.684722 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:12:53.684700    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8\": container with ID starting with 8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8 not found: ID does not exist" containerID="8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8"
Nov 01 21:12:53.684809 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.684734    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8"} err="failed to get container status \"8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8\": rpc error: code = NotFound desc = could not find container \"8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8\": container with ID starting with 8baf628317bb4000bc87ca156b91cdf55f22e24a1432202f1a1635a375a508c8 not found: ID does not exist"
Nov 01 21:12:53.835612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.835520    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"]
Nov 01 21:12:53.838717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:53.838692    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ab520-744969b995-sslqx"]
Nov 01 21:12:55.080109 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:12:55.080071    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d7d2ba-ebef-4765-afb5-e55a6e146e86" path="/var/lib/kubelet/pods/f2d7d2ba-ebef-4765-afb5-e55a6e146e86/volumes"
Nov 01 21:13:15.213130 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:15.213089    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"]
Nov 01 21:13:20.213600 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:20.213529    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerName="payload-pre-processing" containerID="cri-o://53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a" gracePeriod=25
Nov 01 21:13:23.367279 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:13:23.367239    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:13:26.001536 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.001451    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"]
Nov 01 21:13:26.001997 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.001682    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" podUID="a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" containerName="maas-api" containerID="cri-o://f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec" gracePeriod=30
Nov 01 21:13:26.322152 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.322122    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:13:26.492809 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.492767    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca\") pod \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") "
Nov 01 21:13:26.493002 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.492830    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs\") pod \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") "
Nov 01 21:13:26.493002 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.492884    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr\" (UniqueName: \"kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr\") pod \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") "
Nov 01 21:13:26.493341 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.493073    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\") pod \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\" (UID: \"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420\") "
Nov 01 21:13:26.493341 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.493243    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca" pod "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" (UID: "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:13:26.494075 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.494048    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:26.495790 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.495624    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr" pod "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" (UID: "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420"). InnerVolumeSpecName "kube-api-access-xdntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:13:26.495905 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.495812    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls" pod "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" (UID: "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:13:26.496196 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.496168    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs" pod "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" (UID: "a2ae5315-ccc0-4ac7-bc6b-3510df8b1420"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:13:26.596063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.595976    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdntr\" (UniqueName: \"kubernetes.io/projected/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-kube-api-access-xdntr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:26.596063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.596007    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:26.596063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.596018    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:26.794250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.794206    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:13:26.904600 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.904558    4872 generic.go:356] "Generic (PLEG): container finished" podID="a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" containerID="f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec" exitCode=0
Nov 01 21:13:26.904789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.904607    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"
Nov 01 21:13:26.904789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.904646    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" event={"ID":"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420","Type":"ContainerDied","Data":"f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec"}
Nov 01 21:13:26.904789 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.904751    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n" event={"ID":"a2ae5315-ccc0-4ac7-bc6b-3510df8b1420","Type":"ContainerDied","Data":"2d814d7dfd5a7c123d441fd81006b5381eb6066c70c5311bdb73f0b2a1bd63e8"}
Nov 01 21:13:26.904963 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.904789    4872 scope.go:122] "RemoveContainer" containerID="f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec"
Nov 01 21:13:26.919694 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.919666    4872 scope.go:122] "RemoveContainer" containerID="f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec"
Nov 01 21:13:26.920053 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:13:26.920029    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec\": container with ID starting with f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec not found: ID does not exist" containerID="f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec"
Nov 01 21:13:26.920151 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:26.920065    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec"} err="failed to get container status \"f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec\": rpc error: code = NotFound desc = could not find container \"f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec\": container with ID starting with f1f4eac1723c71b8bf2b1c6bc5d461e1c19e8f15146b52d51fd31dbb39f25dec not found: ID does not exist"
Nov 01 21:13:27.091566 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:27.091529    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"]
Nov 01 21:13:27.123947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:27.123915    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b9305-748f4bfcc5-tp29n"]
Nov 01 21:13:29.079668 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:29.079625    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" path="/var/lib/kubelet/pods/a2ae5315-ccc0-4ac7-bc6b-3510df8b1420/volumes"
Nov 01 21:13:36.794729 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:36.794691    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:13:40.498390 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.498359    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 21:13:40.597508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.597468    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume\") pod \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") "
Nov 01 21:13:40.597716 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.597642    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5\" (UniqueName: \"kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5\") pod \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\" (UID: \"162ab2c2-db87-4fb7-b743-2c3c44b6a795\") "
Nov 01 21:13:40.597996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.597966    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume" pod "162ab2c2-db87-4fb7-b743-2c3c44b6a795" (UID: "162ab2c2-db87-4fb7-b743-2c3c44b6a795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:13:40.598222 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.598204    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162ab2c2-db87-4fb7-b743-2c3c44b6a795-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:40.599815 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.599788    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5" pod "162ab2c2-db87-4fb7-b743-2c3c44b6a795" (UID: "162ab2c2-db87-4fb7-b743-2c3c44b6a795"). InnerVolumeSpecName "kube-api-access-xjkx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:13:40.699610 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:40.699572    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkx5\" (UniqueName: \"kubernetes.io/projected/162ab2c2-db87-4fb7-b743-2c3c44b6a795-kube-api-access-xjkx5\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:41.046974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.045162    4872 generic.go:356] "Generic (PLEG): container finished" podID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerID="53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a" exitCode=0
Nov 01 21:13:41.046974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.045353    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" event={"ID":"162ab2c2-db87-4fb7-b743-2c3c44b6a795","Type":"ContainerDied","Data":"53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a"}
Nov 01 21:13:41.046974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.045409    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd" event={"ID":"162ab2c2-db87-4fb7-b743-2c3c44b6a795","Type":"ContainerDied","Data":"398c4cdeca795cc3a37d8cde127645119ba77aef164b82da5fe327e265de83be"}
Nov 01 21:13:41.046974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.045440    4872 scope.go:122] "RemoveContainer" containerID="53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a"
Nov 01 21:13:41.046974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.045678    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"
Nov 01 21:13:41.079077 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.079054    4872 scope.go:122] "RemoveContainer" containerID="53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a"
Nov 01 21:13:41.079395 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:13:41.079368    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a\": container with ID starting with 53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a not found: ID does not exist" containerID="53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a"
Nov 01 21:13:41.079518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.079494    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a"} err="failed to get container status \"53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a\": rpc error: code = NotFound desc = could not find container \"53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a\": container with ID starting with 53a2f3bd19c734410081040ec9b610eb97afb3713a5896e28168e3d0065b537a not found: ID does not exist"
Nov 01 21:13:41.213177 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.213144    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"]
Nov 01 21:13:41.222543 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:41.222488    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cd248f-7b8875bf8c-c8fvd"]
Nov 01 21:13:43.079114 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:43.079074    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" path="/var/lib/kubelet/pods/162ab2c2-db87-4fb7-b743-2c3c44b6a795/volumes"
Nov 01 21:13:47.102976 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:47.102934    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"]
Nov 01 21:13:52.104313 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.104253    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt" podUID="0e39be9c-5579-4fe3-afa0-c99472f7d497" containerName="payload-processing" containerID="cri-o://2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0" gracePeriod=25
Nov 01 21:13:52.391934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.391904    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:13:52.455468 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.455439    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p\" (UniqueName: \"kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p\") pod \"0e39be9c-5579-4fe3-afa0-c99472f7d497\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") "
Nov 01 21:13:52.455637 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.455617    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume\") pod \"0e39be9c-5579-4fe3-afa0-c99472f7d497\" (UID: \"0e39be9c-5579-4fe3-afa0-c99472f7d497\") "
Nov 01 21:13:52.455971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.455941    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume" pod "0e39be9c-5579-4fe3-afa0-c99472f7d497" (UID: "0e39be9c-5579-4fe3-afa0-c99472f7d497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:13:52.457581 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.457558    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p" pod "0e39be9c-5579-4fe3-afa0-c99472f7d497" (UID: "0e39be9c-5579-4fe3-afa0-c99472f7d497"). InnerVolumeSpecName "kube-api-access-hqs9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:13:52.557773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.557737    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqs9p\" (UniqueName: \"kubernetes.io/projected/0e39be9c-5579-4fe3-afa0-c99472f7d497-kube-api-access-hqs9p\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:52.557773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:52.557766    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e39be9c-5579-4fe3-afa0-c99472f7d497-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:13:53.121399 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.121367    4872 generic.go:356] "Generic (PLEG): container finished" podID="0e39be9c-5579-4fe3-afa0-c99472f7d497" containerID="2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0" exitCode=0
Nov 01 21:13:53.121875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.121410    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"
Nov 01 21:13:53.121875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.121448    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt" event={"ID":"0e39be9c-5579-4fe3-afa0-c99472f7d497","Type":"ContainerDied","Data":"2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0"}
Nov 01 21:13:53.121875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.121504    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt" event={"ID":"0e39be9c-5579-4fe3-afa0-c99472f7d497","Type":"ContainerDied","Data":"13264f2b3a8667d1a9b8d5c53e791c9102984e3dc2d08ad790c449ff478aace8"}
Nov 01 21:13:53.121875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.121527    4872 scope.go:122] "RemoveContainer" containerID="2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0"
Nov 01 21:13:53.131291 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.131273    4872 scope.go:122] "RemoveContainer" containerID="2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0"
Nov 01 21:13:53.131578 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:13:53.131559    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0\": container with ID starting with 2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0 not found: ID does not exist" containerID="2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0"
Nov 01 21:13:53.131633 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.131589    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0"} err="failed to get container status \"2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0\": rpc error: code = NotFound desc = could not find container \"2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0\": container with ID starting with 2955405c1d6d8b1ec534d4218891df1139624409ecabdb41bc244d0ae01c2eb0 not found: ID does not exist"
Nov 01 21:13:53.287714 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.287673    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"]
Nov 01 21:13:53.338523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:53.338434    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-044440-5678b676c-cg7nt"]
Nov 01 21:13:53.369788 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:13:53.369753    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:13:55.080060 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:13:55.080026    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e39be9c-5579-4fe3-afa0-c99472f7d497" path="/var/lib/kubelet/pods/0e39be9c-5579-4fe3-afa0-c99472f7d497/volumes"
Nov 01 21:14:20.619187 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:20.619151    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"]
Nov 01 21:14:20.619830 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:20.619414    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" podUID="aa0796ee-20f5-40d4-ac70-98ca2694840c" containerName="maas-api" containerID="cri-o://74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0" gracePeriod=30
Nov 01 21:14:20.936907 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:20.936878    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:14:21.120822 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.120781    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca\") pod \"aa0796ee-20f5-40d4-ac70-98ca2694840c\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") "
Nov 01 21:14:21.121005 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.120917    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") pod \"aa0796ee-20f5-40d4-ac70-98ca2694840c\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") "
Nov 01 21:14:21.121005 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.120982    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t\" (UniqueName: \"kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t\") pod \"aa0796ee-20f5-40d4-ac70-98ca2694840c\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") "
Nov 01 21:14:21.121124 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.121063    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs\") pod \"aa0796ee-20f5-40d4-ac70-98ca2694840c\" (UID: \"aa0796ee-20f5-40d4-ac70-98ca2694840c\") "
Nov 01 21:14:21.121186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.121152    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca" pod "aa0796ee-20f5-40d4-ac70-98ca2694840c" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:14:21.121506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.121478    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0796ee-20f5-40d4-ac70-98ca2694840c-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:21.123071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.123044    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls" pod "aa0796ee-20f5-40d4-ac70-98ca2694840c" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:21.123626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.123606    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t" pod "aa0796ee-20f5-40d4-ac70-98ca2694840c" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c"). InnerVolumeSpecName "kube-api-access-bjp5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:21.123626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.123609    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs" pod "aa0796ee-20f5-40d4-ac70-98ca2694840c" (UID: "aa0796ee-20f5-40d4-ac70-98ca2694840c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:21.223322 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.223225    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:21.223322 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.223260    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0796ee-20f5-40d4-ac70-98ca2694840c-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:21.223322 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.223276    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjp5t\" (UniqueName: \"kubernetes.io/projected/aa0796ee-20f5-40d4-ac70-98ca2694840c-kube-api-access-bjp5t\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:21.295612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.295578    4872 generic.go:356] "Generic (PLEG): container finished" podID="aa0796ee-20f5-40d4-ac70-98ca2694840c" containerID="74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0" exitCode=0
Nov 01 21:14:21.295803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.295621    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"
Nov 01 21:14:21.295803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.295641    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" event={"ID":"aa0796ee-20f5-40d4-ac70-98ca2694840c","Type":"ContainerDied","Data":"74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0"}
Nov 01 21:14:21.295803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.295695    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc" event={"ID":"aa0796ee-20f5-40d4-ac70-98ca2694840c","Type":"ContainerDied","Data":"787f1855b942ca6341c1115571a1281033a616145feb695d638859196815abd5"}
Nov 01 21:14:21.295803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.295717    4872 scope.go:122] "RemoveContainer" containerID="74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0"
Nov 01 21:14:21.306177 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.306153    4872 scope.go:122] "RemoveContainer" containerID="74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0"
Nov 01 21:14:21.306440 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:14:21.306417    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0\": container with ID starting with 74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0 not found: ID does not exist" containerID="74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0"
Nov 01 21:14:21.306515 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.306455    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0"} err="failed to get container status \"74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0\": rpc error: code = NotFound desc = could not find container \"74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0\": container with ID starting with 74d03b294c5407cd785441563ca287ea63fe1fc7b9a30a19a201b45a0d232ef0 not found: ID does not exist"
Nov 01 21:14:21.462436 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.462403    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"]
Nov 01 21:14:21.465471 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:21.465449    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7091fd-7b7788ffb4-wghxc"]
Nov 01 21:14:23.079945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:23.079909    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0796ee-20f5-40d4-ac70-98ca2694840c" path="/var/lib/kubelet/pods/aa0796ee-20f5-40d4-ac70-98ca2694840c/volumes"
Nov 01 21:14:23.366915 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:14:23.366833    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:14:29.359359 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:29.359323    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"]
Nov 01 21:14:29.359982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:29.359574    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy" containerID="cri-o://e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e" gracePeriod=30
Nov 01 21:14:34.640185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.640121    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:14:34.699573 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699544    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699584    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699629    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699719    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699756    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699838    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699880    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699894    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:34.699945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699910    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.699945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.699911    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:34.700230 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700040    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert\") pod \"b11a3ebc-7f1a-4273-a735-821126c88794\" (UID: \"b11a3ebc-7f1a-4273-a735-821126c88794\") "
Nov 01 21:14:34.700230 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700065    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:34.700330 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700238    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:34.700455 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700432    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:14:34.700646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700575    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b11a3ebc-7f1a-4273-a735-821126c88794-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.700646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700598    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.700646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700609    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.700646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700618    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.700646 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.700626    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.702034 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.702011    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:34.702205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.702181    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "kube-api-access-xwxkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:34.702430 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.702408    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:34.702430 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.702410    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo" pod "b11a3ebc-7f1a-4273-a735-821126c88794" (UID: "b11a3ebc-7f1a-4273-a735-821126c88794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:14:34.802411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.802376    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxkf\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-kube-api-access-xwxkf\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.802411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.802406    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b11a3ebc-7f1a-4273-a735-821126c88794-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.802411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.802417    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b11a3ebc-7f1a-4273-a735-821126c88794-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:34.802687 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:34.802426    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b11a3ebc-7f1a-4273-a735-821126c88794-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:35.400521 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.400490    4872 generic.go:356] "Generic (PLEG): container finished" podID="b11a3ebc-7f1a-4273-a735-821126c88794" containerID="e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e" exitCode=0
Nov 01 21:14:35.400736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.400536    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"
Nov 01 21:14:35.400736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.400583    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" event={"ID":"b11a3ebc-7f1a-4273-a735-821126c88794","Type":"ContainerDied","Data":"e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e"}
Nov 01 21:14:35.400736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.400646    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll" event={"ID":"b11a3ebc-7f1a-4273-a735-821126c88794","Type":"ContainerDied","Data":"2e72692ba688e06d9e4ecc219422ff24c93fb286a74551d6a0384fee955fd372"}
Nov 01 21:14:35.400736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.400710    4872 scope.go:122] "RemoveContainer" containerID="e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e"
Nov 01 21:14:35.410811 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.410793    4872 scope.go:122] "RemoveContainer" containerID="e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e"
Nov 01 21:14:35.411110 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:14:35.411088    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e\": container with ID starting with e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e not found: ID does not exist" containerID="e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e"
Nov 01 21:14:35.411168 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.411121    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e"} err="failed to get container status \"e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e\": rpc error: code = NotFound desc = could not find container \"e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e\": container with ID starting with e7966629431135495655cc76fa97cca3b94a4a66323d7e5e3e61ea61ca14c74e not found: ID does not exist"
Nov 01 21:14:35.568791 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.568752    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"]
Nov 01 21:14:35.619505 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:35.619468    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-qk7ll"]
Nov 01 21:14:37.079550 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:37.079517    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" path="/var/lib/kubelet/pods/b11a3ebc-7f1a-4273-a735-821126c88794/volumes"
Nov 01 21:14:43.849801 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:43.849755    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"]
Nov 01 21:14:43.850316 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:43.850055    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" podUID="deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" containerName="maas-api" containerID="cri-o://529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17" gracePeriod=30
Nov 01 21:14:44.162538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.162511    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:14:44.257676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.253903    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs\") pod \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") "
Nov 01 21:14:44.257676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.253971    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4\" (UniqueName: \"kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4\") pod \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") "
Nov 01 21:14:44.257676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.254214    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\") pod \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") "
Nov 01 21:14:44.257676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.254267    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca\") pod \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\" (UID: \"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b\") "
Nov 01 21:14:44.258055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.257819    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca" pod "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" (UID: "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:14:44.264035 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.263993    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4" pod "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" (UID: "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b"). InnerVolumeSpecName "kube-api-access-qxnl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:44.273392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.265691    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs" pod "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" (UID: "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:44.273392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.266404    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnl4\" (UniqueName: \"kubernetes.io/projected/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-kube-api-access-qxnl4\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:44.273392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.267624    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:44.280818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.280775    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls" pod "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" (UID: "deccf8ce-cbdd-41f1-b1a8-f105b0394e7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:44.369414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.369372    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:44.369414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.369414    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:44.456757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.456722    4872 generic.go:356] "Generic (PLEG): container finished" podID="deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" containerID="529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17" exitCode=0
Nov 01 21:14:44.456936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.456765    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" event={"ID":"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b","Type":"ContainerDied","Data":"529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17"}
Nov 01 21:14:44.456936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.456786    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"
Nov 01 21:14:44.456936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.456838    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf" event={"ID":"deccf8ce-cbdd-41f1-b1a8-f105b0394e7b","Type":"ContainerDied","Data":"ef6b88afcf5c008705628388335b9ed710d59d83c4e566fcca3c8863eba2bb89"}
Nov 01 21:14:44.456936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.456867    4872 scope.go:122] "RemoveContainer" containerID="529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17"
Nov 01 21:14:44.467028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.467010    4872 scope.go:122] "RemoveContainer" containerID="529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17"
Nov 01 21:14:44.467294 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:14:44.467273    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17\": container with ID starting with 529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17 not found: ID does not exist" containerID="529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17"
Nov 01 21:14:44.467382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.467308    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17"} err="failed to get container status \"529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17\": rpc error: code = NotFound desc = could not find container \"529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17\": container with ID starting with 529a1dc3925add0523e360d46f9aff480784cbe4fecbbc0e00ecf1b6c8469f17 not found: ID does not exist"
Nov 01 21:14:44.639677 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.631973    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"]
Nov 01 21:14:44.642696 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:44.641161    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-044440-65d44d9979-xh7bf"]
Nov 01 21:14:45.080154 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:45.080120    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" path="/var/lib/kubelet/pods/deccf8ce-cbdd-41f1-b1a8-f105b0394e7b/volumes"
Nov 01 21:14:53.374876 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:14:53.374837    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:14:54.487025 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:54.486989    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"]
Nov 01 21:14:55.531207 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.531128    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"]
Nov 01 21:14:55.531613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.531312    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerName="discovery" containerID="cri-o://fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02" gracePeriod=30
Nov 01 21:14:55.813253 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.813223    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 21:14:55.921823 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.921788    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.921977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.921840    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.921977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.921867    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.921977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.921963    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.922126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.922095    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.922179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.922147    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.922244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.922174    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs\") pod \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\" (UID: \"b479970e-3596-4ee6-bbd2-700dbdcf0d3f\") "
Nov 01 21:14:55.922723 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.922684    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:14:55.922888 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.922862    4872 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-ca-configmap\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.924574 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.924547    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:55.924808 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.924781    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:55.924808 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.924791    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:55.924981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.924824    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "kube-api-access-vhc9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:55.924981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.924823    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:55.924981 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:55.924859    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig" pod "b479970e-3596-4ee6-bbd2-700dbdcf0d3f" (UID: "b479970e-3596-4ee6-bbd2-700dbdcf0d3f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:14:56.023636 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.023600    4872 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-csr-dns-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:56.023636 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.023629    4872 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-kubeconfig\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:56.023636 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.023640    4872 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-local-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:56.023888 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.023649    4872 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-cacerts\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:56.023888 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.023671    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:56.023888 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.023681    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhc9f\" (UniqueName: \"kubernetes.io/projected/b479970e-3596-4ee6-bbd2-700dbdcf0d3f-kube-api-access-vhc9f\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:56.536197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.536160    4872 generic.go:356] "Generic (PLEG): container finished" podID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerID="fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02" exitCode=0
Nov 01 21:14:56.536717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.536212    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"
Nov 01 21:14:56.536717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.536242    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" event={"ID":"b479970e-3596-4ee6-bbd2-700dbdcf0d3f","Type":"ContainerDied","Data":"fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02"}
Nov 01 21:14:56.536717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.536307    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t" event={"ID":"b479970e-3596-4ee6-bbd2-700dbdcf0d3f","Type":"ContainerDied","Data":"91f2fc3b8e98e9e31a4e4eeffc057e301ad9ad2dfd6ee0ed08caa4b7faaf1e5a"}
Nov 01 21:14:56.536717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.536337    4872 scope.go:122] "RemoveContainer" containerID="fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02"
Nov 01 21:14:56.547302 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.547286    4872 scope.go:122] "RemoveContainer" containerID="fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02"
Nov 01 21:14:56.547522 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:14:56.547500    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02\": container with ID starting with fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02 not found: ID does not exist" containerID="fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02"
Nov 01 21:14:56.547584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.547532    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02"} err="failed to get container status \"fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02\": rpc error: code = NotFound desc = could not find container \"fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02\": container with ID starting with fa6a0a6015740b5382814c8c33ff59f503040bf3abbbf4f6bed246b965089c02 not found: ID does not exist"
Nov 01 21:14:56.703195 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.703116    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"]
Nov 01 21:14:56.754613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:56.754574    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmd5t"]
Nov 01 21:14:57.079967 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:57.079928    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" path="/var/lib/kubelet/pods/b479970e-3596-4ee6-bbd2-700dbdcf0d3f/volumes"
Nov 01 21:14:59.487987 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.487942    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck" podUID="6988e2ad-3e8b-433c-9491-3035033a73da" containerName="payload-processing" containerID="cri-o://39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966" gracePeriod=25
Nov 01 21:14:59.752979 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.752916    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:14:59.872625 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.872573    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume\") pod \"6988e2ad-3e8b-433c-9491-3035033a73da\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") "
Nov 01 21:14:59.872835 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.872809    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m\" (UniqueName: \"kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m\") pod \"6988e2ad-3e8b-433c-9491-3035033a73da\" (UID: \"6988e2ad-3e8b-433c-9491-3035033a73da\") "
Nov 01 21:14:59.873350 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.873316    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume" pod "6988e2ad-3e8b-433c-9491-3035033a73da" (UID: "6988e2ad-3e8b-433c-9491-3035033a73da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:14:59.875900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.875867    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m" pod "6988e2ad-3e8b-433c-9491-3035033a73da" (UID: "6988e2ad-3e8b-433c-9491-3035033a73da"). InnerVolumeSpecName "kube-api-access-mh28m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:59.974944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.974913    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh28m\" (UniqueName: \"kubernetes.io/projected/6988e2ad-3e8b-433c-9491-3035033a73da-kube-api-access-mh28m\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:59.974944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:14:59.974941    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988e2ad-3e8b-433c-9491-3035033a73da-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:00.188922 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.188886    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"]
Nov 01 21:15:00.189399 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189383    4872 cpu_manager.go:414] "containerMap: removing container" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerName="discovery"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189399    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerName="discovery"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189411    4872 cpu_manager.go:414] "containerMap: removing container" podUID="a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" containerName="maas-api"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189416    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" containerName="maas-api"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189431    4872 cpu_manager.go:414] "containerMap: removing container" podUID="deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" containerName="maas-api"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189437    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" containerName="maas-api"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189446    4872 cpu_manager.go:414] "containerMap: removing container" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy"
Nov 01 21:15:00.189464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189451    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189479    4872 cpu_manager.go:414] "containerMap: removing container" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189484    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189492    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f2d7d2ba-ebef-4765-afb5-e55a6e146e86" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189497    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d7d2ba-ebef-4765-afb5-e55a6e146e86" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189503    4872 cpu_manager.go:414] "containerMap: removing container" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerName="payload-pre-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189508    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerName="payload-pre-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189525    4872 cpu_manager.go:414] "containerMap: removing container" podUID="0e39be9c-5579-4fe3-afa0-c99472f7d497" containerName="payload-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189530    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e39be9c-5579-4fe3-afa0-c99472f7d497" containerName="payload-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189537    4872 cpu_manager.go:414] "containerMap: removing container" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189541    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189547    4872 cpu_manager.go:414] "containerMap: removing container" podUID="6988e2ad-3e8b-433c-9491-3035033a73da" containerName="payload-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189551    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6988e2ad-3e8b-433c-9491-3035033a73da" containerName="payload-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189565    4872 cpu_manager.go:414] "containerMap: removing container" podUID="aa0796ee-20f5-40d4-ac70-98ca2694840c" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189569    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0796ee-20f5-40d4-ac70-98ca2694840c" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189646    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e39be9c-5579-4fe3-afa0-c99472f7d497" containerName="payload-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189665    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11a3ebc-7f1a-4273-a735-821126c88794" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189672    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="6988e2ad-3e8b-433c-9491-3035033a73da" containerName="payload-processing"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189680    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0796ee-20f5-40d4-ac70-98ca2694840c" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189688    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="b479970e-3596-4ee6-bbd2-700dbdcf0d3f" containerName="discovery"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189694    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ae5315-ccc0-4ac7-bc6b-3510df8b1420" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189701    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="deccf8ce-cbdd-41f1-b1a8-f105b0394e7b" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189706    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189712    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d37b7fd-beb4-428c-8f6b-ca28097b15a9" containerName="istio-proxy"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189717    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d7d2ba-ebef-4765-afb5-e55a6e146e86" containerName="maas-api"
Nov 01 21:15:00.189818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.189723    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="162ab2c2-db87-4fb7-b743-2c3c44b6a795" containerName="payload-pre-processing"
Nov 01 21:15:00.192974 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.192957    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.205364 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.205336    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"]
Nov 01 21:15:00.209364 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.209340    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.246187 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.246156    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\""
Nov 01 21:15:00.277438 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.277404    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892795-tzdx4\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.277619 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.277471    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.277619 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.277504    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct75t\" (UniqueName: \"kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t\") pod \"maas-api-key-cleanup-29892795-tzdx4\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.277805 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.277638    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qwb\" (UniqueName: \"kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.320966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.320933    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"]
Nov 01 21:15:00.379145 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.379110    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892795-tzdx4\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.379471 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.379167    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.379471 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.379284    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct75t\" (UniqueName: \"kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t\") pod \"maas-api-key-cleanup-29892795-tzdx4\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.379471 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.379356    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qwb\" (UniqueName: \"kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.379963 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.379940    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.380060 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.379943    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892795-tzdx4\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.388901 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.388869    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"]
Nov 01 21:15:00.436286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.436248    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct75t\" (UniqueName: \"kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t\") pod \"maas-api-key-cleanup-29892795-tzdx4\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.436286 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.436250    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qwb\" (UniqueName: \"kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.508287 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.508206    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:00.522622 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.522592    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:00.571474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.571389    4872 generic.go:356] "Generic (PLEG): container finished" podID="6988e2ad-3e8b-433c-9491-3035033a73da" containerID="39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966" exitCode=0
Nov 01 21:15:00.571563 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.571527    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck" event={"ID":"6988e2ad-3e8b-433c-9491-3035033a73da","Type":"ContainerDied","Data":"39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966"}
Nov 01 21:15:00.571628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.571588    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck" event={"ID":"6988e2ad-3e8b-433c-9491-3035033a73da","Type":"ContainerDied","Data":"268104471a7c357945a044ba5b0325a02eb8f80525d752cda969cdd74c3632d6"}
Nov 01 21:15:00.571628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.571618    4872 scope.go:122] "RemoveContainer" containerID="39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966"
Nov 01 21:15:00.571821 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.571804    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"
Nov 01 21:15:00.589052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.589029    4872 scope.go:122] "RemoveContainer" containerID="39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966"
Nov 01 21:15:00.589469 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:00.589417    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966\": container with ID starting with 39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966 not found: ID does not exist" containerID="39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966"
Nov 01 21:15:00.589469 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.589449    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966"} err="failed to get container status \"39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966\": rpc error: code = NotFound desc = could not find container \"39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966\": container with ID starting with 39c721f8166da878e8de8d3012a25b8b1368ab728a4892ec0eb3f7dafd61b966 not found: ID does not exist"
Nov 01 21:15:00.739308 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.739269    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"]
Nov 01 21:15:00.746416 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.746381    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-670df9-86c5df874f-vm2ck"]
Nov 01 21:15:00.889262 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.889235    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"]
Nov 01 21:15:00.891081 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:15:00.891053    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0184315a_2824_4828_a824_a6ed7438150d.slice/crio-3b1daa01e879783f63cfda91f300c9a001fd47a606469f949afde0fef7f809b6 WatchSource:0}: Error finding container 3b1daa01e879783f63cfda91f300c9a001fd47a606469f949afde0fef7f809b6: Status 404 returned error can't find the container with id 3b1daa01e879783f63cfda91f300c9a001fd47a606469f949afde0fef7f809b6
Nov 01 21:15:00.939272 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:00.939249    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"]
Nov 01 21:15:00.941635 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:15:00.941606    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8125de3_d3f2_40c2_a169_9a733fadaf1b.slice/crio-8376b5d503f66a20035adf65a654c1b57d68005e5dd42a14f23c9b14d9ba9486 WatchSource:0}: Error finding container 8376b5d503f66a20035adf65a654c1b57d68005e5dd42a14f23c9b14d9ba9486: Status 404 returned error can't find the container with id 8376b5d503f66a20035adf65a654c1b57d68005e5dd42a14f23c9b14d9ba9486
Nov 01 21:15:01.079882 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.079846    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6988e2ad-3e8b-433c-9491-3035033a73da" path="/var/lib/kubelet/pods/6988e2ad-3e8b-433c-9491-3035033a73da/volumes"
Nov 01 21:15:01.579699 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.579585    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerStarted","Data":"bba8f026440c0d020e3b6b312df52c63b69cb545f1fc42340567756bbdff371b"}
Nov 01 21:15:01.580165 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.579713    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerStarted","Data":"3b1daa01e879783f63cfda91f300c9a001fd47a606469f949afde0fef7f809b6"}
Nov 01 21:15:01.581224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.581190    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerStarted","Data":"abf9d4567f45133e5af116d85978ff1e8ecabe643b6a521eb619c91f7a8a32a1"}
Nov 01 21:15:01.581357 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.581248    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerStarted","Data":"8376b5d503f66a20035adf65a654c1b57d68005e5dd42a14f23c9b14d9ba9486"}
Nov 01 21:15:01.693165 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.693109    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" podStartSLOduration=1.693096847 podStartE2EDuration="1.693096847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:01.692941641 +0000 UTC m=+2658.742051461" watchObservedRunningTime="2026-11-01 21:15:01.693096847 +0000 UTC m=+2658.742206665"
Nov 01 21:15:01.808363 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:01.808312    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" podStartSLOduration=1.808294521 podStartE2EDuration="1.808294521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:01.805955954 +0000 UTC m=+2658.855065783" watchObservedRunningTime="2026-11-01 21:15:01.808294521 +0000 UTC m=+2658.857404339"
Nov 01 21:15:02.589520 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:02.589484    4872 generic.go:356] "Generic (PLEG): container finished" podID="0184315a-2824-4828-a824-a6ed7438150d" containerID="bba8f026440c0d020e3b6b312df52c63b69cb545f1fc42340567756bbdff371b" exitCode=60
Nov 01 21:15:02.590003 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:02.589567    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerDied","Data":"bba8f026440c0d020e3b6b312df52c63b69cb545f1fc42340567756bbdff371b"}
Nov 01 21:15:02.590003 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:02.589920    4872 scope.go:122] "RemoveContainer" containerID="bba8f026440c0d020e3b6b312df52c63b69cb545f1fc42340567756bbdff371b"
Nov 01 21:15:02.591209 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:02.591185    4872 generic.go:356] "Generic (PLEG): container finished" podID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerID="abf9d4567f45133e5af116d85978ff1e8ecabe643b6a521eb619c91f7a8a32a1" exitCode=60
Nov 01 21:15:02.591308 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:02.591218    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerDied","Data":"abf9d4567f45133e5af116d85978ff1e8ecabe643b6a521eb619c91f7a8a32a1"}
Nov 01 21:15:02.591465 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:02.591450    4872 scope.go:122] "RemoveContainer" containerID="abf9d4567f45133e5af116d85978ff1e8ecabe643b6a521eb619c91f7a8a32a1"
Nov 01 21:15:03.599644 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.599611    4872 generic.go:356] "Generic (PLEG): container finished" podID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerID="df69631df0182eace4eb692a36d073d73482393fa8c2a777fb3d6a326ac0e3da" exitCode=60
Nov 01 21:15:03.600116 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.599701    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerDied","Data":"df69631df0182eace4eb692a36d073d73482393fa8c2a777fb3d6a326ac0e3da"}
Nov 01 21:15:03.600116 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.599762    4872 scope.go:122] "RemoveContainer" containerID="abf9d4567f45133e5af116d85978ff1e8ecabe643b6a521eb619c91f7a8a32a1"
Nov 01 21:15:03.600230 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.600117    4872 scope.go:122] "RemoveContainer" containerID="df69631df0182eace4eb692a36d073d73482393fa8c2a777fb3d6a326ac0e3da"
Nov 01 21:15:03.600486 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:03.600460    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg_odh-ai-gateway-infra(c8125de3-d3f2-40c2-a169-9a733fadaf1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b"
Nov 01 21:15:03.601785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.601762    4872 generic.go:356] "Generic (PLEG): container finished" podID="0184315a-2824-4828-a824-a6ed7438150d" containerID="2adede06021e5576fb2093094e030f041809d4f86af84032f516527550d3bbd4" exitCode=60
Nov 01 21:15:03.601860 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.601837    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerDied","Data":"2adede06021e5576fb2093094e030f041809d4f86af84032f516527550d3bbd4"}
Nov 01 21:15:03.602112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.602099    4872 scope.go:122] "RemoveContainer" containerID="2adede06021e5576fb2093094e030f041809d4f86af84032f516527550d3bbd4"
Nov 01 21:15:03.602363 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:03.602347    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892795-tzdx4_odh-ai-gateway-infra(0184315a-2824-4828-a824-a6ed7438150d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" podUID="0184315a-2824-4828-a824-a6ed7438150d"
Nov 01 21:15:03.633198 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:03.633169    4872 scope.go:122] "RemoveContainer" containerID="bba8f026440c0d020e3b6b312df52c63b69cb545f1fc42340567756bbdff371b"
Nov 01 21:15:04.609955 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:04.609929    4872 scope.go:122] "RemoveContainer" containerID="2adede06021e5576fb2093094e030f041809d4f86af84032f516527550d3bbd4"
Nov 01 21:15:04.610444 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:04.610310    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892795-tzdx4_odh-ai-gateway-infra(0184315a-2824-4828-a824-a6ed7438150d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" podUID="0184315a-2824-4828-a824-a6ed7438150d"
Nov 01 21:15:04.611558 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:04.611543    4872 scope.go:122] "RemoveContainer" containerID="df69631df0182eace4eb692a36d073d73482393fa8c2a777fb3d6a326ac0e3da"
Nov 01 21:15:04.611824 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:04.611807    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg_odh-ai-gateway-infra(c8125de3-d3f2-40c2-a169-9a733fadaf1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b"
Nov 01 21:15:10.367101 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.367062    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"]
Nov 01 21:15:10.367580 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.367288    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" podUID="f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" containerName="maas-api" containerID="cri-o://c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722" gracePeriod=30
Nov 01 21:15:10.639357 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.639331    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 21:15:10.645203 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.645156    4872 generic.go:356] "Generic (PLEG): container finished" podID="f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" containerID="c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722" exitCode=0
Nov 01 21:15:10.645203 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.645192    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" event={"ID":"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1","Type":"ContainerDied","Data":"c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722"}
Nov 01 21:15:10.645434 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.645225    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"
Nov 01 21:15:10.645434 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.645259    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2" event={"ID":"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1","Type":"ContainerDied","Data":"5bfc52228a998d1fada919e6bef9efb09d9f8b517faf42244959b6603bddf083"}
Nov 01 21:15:10.645434 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.645280    4872 scope.go:122] "RemoveContainer" containerID="c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722"
Nov 01 21:15:10.655399 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.655378    4872 scope.go:122] "RemoveContainer" containerID="c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722"
Nov 01 21:15:10.655725 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:10.655704    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722\": container with ID starting with c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722 not found: ID does not exist" containerID="c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722"
Nov 01 21:15:10.655779 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.655750    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722"} err="failed to get container status \"c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722\": rpc error: code = NotFound desc = could not find container \"c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722\": container with ID starting with c2dd2b19062d12bffccf1e8eaba9907f6d7171a9e801f118f3c5cf36a74fd722 not found: ID does not exist"
Nov 01 21:15:10.712283 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.712245    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\") pod \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") "
Nov 01 21:15:10.712462 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.712377    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs\") pod \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") "
Nov 01 21:15:10.712510 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.712481    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca\") pod \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") "
Nov 01 21:15:10.712510 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.712498    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25\" (UniqueName: \"kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25\") pod \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\" (UID: \"f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1\") "
Nov 01 21:15:10.712893 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.712862    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca" pod "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" (UID: "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:10.714484 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.714461    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs" pod "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" (UID: "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:10.714583 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.714521    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls" pod "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" (UID: "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:10.714696 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.714643    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25" pod "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" (UID: "f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1"). InnerVolumeSpecName "kube-api-access-k2m25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:10.813546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.813513    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:10.813546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.813542    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2m25\" (UniqueName: \"kubernetes.io/projected/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-kube-api-access-k2m25\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:10.813546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.813552    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:10.813546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:10.813563    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:11.067883 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:11.067842    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"]
Nov 01 21:15:11.078783 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:11.078754    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cd248f-576d88b876-thzq2"]
Nov 01 21:15:13.079593 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:13.079559    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" path="/var/lib/kubelet/pods/f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1/volumes"
Nov 01 21:15:15.071989 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:15.071951    4872 scope.go:122] "RemoveContainer" containerID="2adede06021e5576fb2093094e030f041809d4f86af84032f516527550d3bbd4"
Nov 01 21:15:15.072490 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:15.072036    4872 scope.go:122] "RemoveContainer" containerID="df69631df0182eace4eb692a36d073d73482393fa8c2a777fb3d6a326ac0e3da"
Nov 01 21:15:15.677621 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:15.677584    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerStarted","Data":"378eb25aadb92a2f7959e53e1a6ca1c299f1c3a4073a9ff0d0ab288ee26886b8"}
Nov 01 21:15:15.679542 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:15.679519    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerStarted","Data":"a169d958d8c9255126bdb7525b8129678a77c7e66e2919d7cf513878a952c0db"}
Nov 01 21:15:16.687225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.687192    4872 generic.go:356] "Generic (PLEG): container finished" podID="0184315a-2824-4828-a824-a6ed7438150d" containerID="a169d958d8c9255126bdb7525b8129678a77c7e66e2919d7cf513878a952c0db" exitCode=60
Nov 01 21:15:16.687739 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.687269    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerDied","Data":"a169d958d8c9255126bdb7525b8129678a77c7e66e2919d7cf513878a952c0db"}
Nov 01 21:15:16.687739 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.687334    4872 scope.go:122] "RemoveContainer" containerID="2adede06021e5576fb2093094e030f041809d4f86af84032f516527550d3bbd4"
Nov 01 21:15:16.687739 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.687710    4872 scope.go:122] "RemoveContainer" containerID="a169d958d8c9255126bdb7525b8129678a77c7e66e2919d7cf513878a952c0db"
Nov 01 21:15:16.688063 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:16.688035    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892795-tzdx4_odh-ai-gateway-infra(0184315a-2824-4828-a824-a6ed7438150d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" podUID="0184315a-2824-4828-a824-a6ed7438150d"
Nov 01 21:15:16.689437 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.689416    4872 generic.go:356] "Generic (PLEG): container finished" podID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerID="378eb25aadb92a2f7959e53e1a6ca1c299f1c3a4073a9ff0d0ab288ee26886b8" exitCode=60
Nov 01 21:15:16.689514 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.689466    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerDied","Data":"378eb25aadb92a2f7959e53e1a6ca1c299f1c3a4073a9ff0d0ab288ee26886b8"}
Nov 01 21:15:16.689788 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.689773    4872 scope.go:122] "RemoveContainer" containerID="378eb25aadb92a2f7959e53e1a6ca1c299f1c3a4073a9ff0d0ab288ee26886b8"
Nov 01 21:15:16.690010 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:16.689994    4872 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg_odh-ai-gateway-infra(c8125de3-d3f2-40c2-a169-9a733fadaf1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b"
Nov 01 21:15:16.699698 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.699677    4872 scope.go:122] "RemoveContainer" containerID="df69631df0182eace4eb692a36d073d73482393fa8c2a777fb3d6a326ac0e3da"
Nov 01 21:15:16.802284 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.802249    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"]
Nov 01 21:15:16.916006 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:16.915972    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"]
Nov 01 21:15:17.895940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.895909    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:17.898128 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.898108    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:17.999474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.999381    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb\" (UniqueName: \"kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb\") pod \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") "
Nov 01 21:15:17.999474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.999426    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca\") pod \"0184315a-2824-4828-a824-a6ed7438150d\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") "
Nov 01 21:15:17.999747 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.999536    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t\" (UniqueName: \"kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t\") pod \"0184315a-2824-4828-a824-a6ed7438150d\" (UID: \"0184315a-2824-4828-a824-a6ed7438150d\") "
Nov 01 21:15:17.999747 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.999615    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca\") pod \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\" (UID: \"c8125de3-d3f2-40c2-a169-9a733fadaf1b\") "
Nov 01 21:15:17.999942 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:17.999910    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca" pod "0184315a-2824-4828-a824-a6ed7438150d" (UID: "0184315a-2824-4828-a824-a6ed7438150d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:18.000036 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.000011    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca" pod "c8125de3-d3f2-40c2-a169-9a733fadaf1b" (UID: "c8125de3-d3f2-40c2-a169-9a733fadaf1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:18.000117 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.000099    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0184315a-2824-4828-a824-a6ed7438150d-maas-api-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:18.001707 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.001685    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb" pod "c8125de3-d3f2-40c2-a169-9a733fadaf1b" (UID: "c8125de3-d3f2-40c2-a169-9a733fadaf1b"). InnerVolumeSpecName "kube-api-access-t8qwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:18.001905 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.001883    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t" pod "0184315a-2824-4828-a824-a6ed7438150d" (UID: "0184315a-2824-4828-a824-a6ed7438150d"). InnerVolumeSpecName "kube-api-access-ct75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:18.101395 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.101359    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8qwb\" (UniqueName: \"kubernetes.io/projected/c8125de3-d3f2-40c2-a169-9a733fadaf1b-kube-api-access-t8qwb\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:18.101395 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.101389    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct75t\" (UniqueName: \"kubernetes.io/projected/0184315a-2824-4828-a824-a6ed7438150d-kube-api-access-ct75t\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:18.101395 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.101399    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8125de3-d3f2-40c2-a169-9a733fadaf1b-maas-api-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:18.706708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.706671    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"
Nov 01 21:15:18.706708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.706688    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg" event={"ID":"c8125de3-d3f2-40c2-a169-9a733fadaf1b","Type":"ContainerDied","Data":"8376b5d503f66a20035adf65a654c1b57d68005e5dd42a14f23c9b14d9ba9486"}
Nov 01 21:15:18.706973 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.706762    4872 scope.go:122] "RemoveContainer" containerID="378eb25aadb92a2f7959e53e1a6ca1c299f1c3a4073a9ff0d0ab288ee26886b8"
Nov 01 21:15:18.708059 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.707991    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4" event={"ID":"0184315a-2824-4828-a824-a6ed7438150d","Type":"ContainerDied","Data":"3b1daa01e879783f63cfda91f300c9a001fd47a606469f949afde0fef7f809b6"}
Nov 01 21:15:18.708059 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.708018    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"
Nov 01 21:15:18.716690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.716666    4872 scope.go:122] "RemoveContainer" containerID="a169d958d8c9255126bdb7525b8129678a77c7e66e2919d7cf513878a952c0db"
Nov 01 21:15:18.872359 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.872318    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"]
Nov 01 21:15:18.876401 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:18.876369    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-tzdx4"]
Nov 01 21:15:19.012628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:19.012521    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"]
Nov 01 21:15:19.015813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:19.015781    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892795-5b7gg"]
Nov 01 21:15:19.080946 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:19.080912    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0184315a-2824-4828-a824-a6ed7438150d" path="/var/lib/kubelet/pods/0184315a-2824-4828-a824-a6ed7438150d/volumes"
Nov 01 21:15:19.081388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:19.081374    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" path="/var/lib/kubelet/pods/c8125de3-d3f2-40c2-a169-9a733fadaf1b/volumes"
Nov 01 21:15:23.381265 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:23.381194    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:15:30.329647 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:30.329614    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"]
Nov 01 21:15:30.330055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:30.329832    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy" containerID="cri-o://0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f" gracePeriod=30
Nov 01 21:15:35.593559 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.593532    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 21:15:35.730772 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730736    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.730772 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730781    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730813    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730842    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730900    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730940    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.730980    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731047    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731078    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:35.731292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731120    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs\") pod \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\" (UID: \"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f\") "
Nov 01 21:15:35.731292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731242    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:35.731292 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731259    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:35.731586 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731560    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:35.731650 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731625    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:35.731805 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731790    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.731872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731806    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.731872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731815    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.731872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731824    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.731872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.731834    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.733461 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.733438    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:35.733461 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.733452    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:35.733642 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.733567    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:15:35.733642 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.733568    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n" pod "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" (UID: "1342e2c8-5e98-48e3-8e34-3ea77dc9c74f"). InnerVolumeSpecName "kube-api-access-pxk6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:35.818021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.817984    4872 generic.go:356] "Generic (PLEG): container finished" podID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerID="0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f" exitCode=0
Nov 01 21:15:35.818224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.818061    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"
Nov 01 21:15:35.818224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.818063    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" event={"ID":"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f","Type":"ContainerDied","Data":"0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f"}
Nov 01 21:15:35.818224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.818141    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm" event={"ID":"1342e2c8-5e98-48e3-8e34-3ea77dc9c74f","Type":"ContainerDied","Data":"3525ffc64c08eb1ef89738d7a2b93342396c7105955ab2958c2c99e35dde0ac9"}
Nov 01 21:15:35.818224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.818163    4872 scope.go:122] "RemoveContainer" containerID="0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f"
Nov 01 21:15:35.830166 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.830143    4872 scope.go:122] "RemoveContainer" containerID="1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"
Nov 01 21:15:35.833149 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.833122    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxk6n\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-kube-api-access-pxk6n\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.833260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.833151    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.833260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.833165    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.833260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.833178    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:35.848628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.848596    4872 scope.go:122] "RemoveContainer" containerID="0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f"
Nov 01 21:15:35.849947 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:35.849063    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f\": container with ID starting with 0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f not found: ID does not exist" containerID="0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f"
Nov 01 21:15:35.849947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.849107    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f"} err="failed to get container status \"0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f\": rpc error: code = NotFound desc = could not find container \"0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f\": container with ID starting with 0d33cfb859a9769792c9b0dcda64c920eb68f4fb1880994aabaa6db24dc9767f not found: ID does not exist"
Nov 01 21:15:35.849947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.849133    4872 scope.go:122] "RemoveContainer" containerID="1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"
Nov 01 21:15:35.849947 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:35.849454    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde\": container with ID starting with 1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde not found: ID does not exist" containerID="1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"
Nov 01 21:15:35.849947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.849483    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde"} err="failed to get container status \"1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde\": rpc error: code = NotFound desc = could not find container \"1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde\": container with ID starting with 1d1e9565fadb4d0d5ff1ef3ed197ec63498cdbdc841072789c28d89a8bf6ddde not found: ID does not exist"
Nov 01 21:15:35.984483 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.984392    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"]
Nov 01 21:15:35.991412 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:35.991377    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dce296-openshift-default-84fd9bf7ff-htbpm"]
Nov 01 21:15:37.079280 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:37.079250    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" path="/var/lib/kubelet/pods/1342e2c8-5e98-48e3-8e34-3ea77dc9c74f/volumes"
Nov 01 21:15:37.584387 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:37.584348    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"]
Nov 01 21:15:37.633256 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:37.633225    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"]
Nov 01 21:15:37.635686 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:37.635649    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"]
Nov 01 21:15:43.580310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:43.580273    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:15:43.582754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:43.582730    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:15:43.586001 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:43.585976    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:15:43.588030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:43.588007    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:15:46.099547 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.099506    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"]
Nov 01 21:15:46.100111 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.099741    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" podUID="66f7599d-77fe-41d6-a9e6-2be4a2515e02" containerName="maas-api" containerID="cri-o://a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537" gracePeriod=30
Nov 01 21:15:46.371342 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.371317    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 21:15:46.472449 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.471279    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs\") pod \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") "
Nov 01 21:15:46.472449 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.471362    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs\" (UniqueName: \"kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs\") pod \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") "
Nov 01 21:15:46.472449 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.471506    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\") pod \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") "
Nov 01 21:15:46.472449 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.471718    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca\") pod \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\" (UID: \"66f7599d-77fe-41d6-a9e6-2be4a2515e02\") "
Nov 01 21:15:46.474920 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.473506    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca" pod "66f7599d-77fe-41d6-a9e6-2be4a2515e02" (UID: "66f7599d-77fe-41d6-a9e6-2be4a2515e02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:46.474920 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.474875    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs" pod "66f7599d-77fe-41d6-a9e6-2be4a2515e02" (UID: "66f7599d-77fe-41d6-a9e6-2be4a2515e02"). InnerVolumeSpecName "kube-api-access-hhwxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:46.475406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.475366    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs" pod "66f7599d-77fe-41d6-a9e6-2be4a2515e02" (UID: "66f7599d-77fe-41d6-a9e6-2be4a2515e02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:46.475726 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.475702    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f7599d-77fe-41d6-a9e6-2be4a2515e02-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:46.475957 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.475731    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:46.475957 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.475747    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwxs\" (UniqueName: \"kubernetes.io/projected/66f7599d-77fe-41d6-a9e6-2be4a2515e02-kube-api-access-hhwxs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:46.475957 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.475776    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls" pod "66f7599d-77fe-41d6-a9e6-2be4a2515e02" (UID: "66f7599d-77fe-41d6-a9e6-2be4a2515e02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:46.577166 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.577135    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f7599d-77fe-41d6-a9e6-2be4a2515e02-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:46.904337 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.904271    4872 generic.go:356] "Generic (PLEG): container finished" podID="66f7599d-77fe-41d6-a9e6-2be4a2515e02" containerID="a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537" exitCode=0
Nov 01 21:15:46.905611 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.904580    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" event={"ID":"66f7599d-77fe-41d6-a9e6-2be4a2515e02","Type":"ContainerDied","Data":"a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537"}
Nov 01 21:15:46.905611 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.904639    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v" event={"ID":"66f7599d-77fe-41d6-a9e6-2be4a2515e02","Type":"ContainerDied","Data":"93a4cbfd0ac4de852924d25f6b7110f4d8b765c73ed01589abf40f1b95b48699"}
Nov 01 21:15:46.905611 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.904702    4872 scope.go:122] "RemoveContainer" containerID="a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537"
Nov 01 21:15:46.905611 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.904871    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"
Nov 01 21:15:46.931593 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.931565    4872 scope.go:122] "RemoveContainer" containerID="a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537"
Nov 01 21:15:46.931946 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:46.931921    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537\": container with ID starting with a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537 not found: ID does not exist" containerID="a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537"
Nov 01 21:15:46.932023 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:46.931953    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537"} err="failed to get container status \"a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537\": rpc error: code = NotFound desc = could not find container \"a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537\": container with ID starting with a96ed1b01c4e78a1d4df7a0eb83aaa3af6643cce43897558d71cdf05e6943537 not found: ID does not exist"
Nov 01 21:15:47.092563 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:47.092522    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"]
Nov 01 21:15:47.095081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:47.095058    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24f923-dbd4c68cf-7cd4v"]
Nov 01 21:15:49.078698 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:49.078645    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f7599d-77fe-41d6-a9e6-2be4a2515e02" path="/var/lib/kubelet/pods/66f7599d-77fe-41d6-a9e6-2be4a2515e02/volumes"
Nov 01 21:15:49.418234 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:49.418201    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"]
Nov 01 21:15:49.468753 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:49.468720    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"]
Nov 01 21:15:52.605696 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.605630    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" containerID="cri-o://08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803" gracePeriod=45
Nov 01 21:15:52.653652 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.653603    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main" containerID="cri-o://80f7526d9c96fbc3e2623a532e163de9d03949b43fb5eb8b3d6df57c0775fa70" gracePeriod=45
Nov 01 21:15:52.655208 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.655134    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" containerID="cri-o://5056f5e519d4af34b9253656d87b518c7390fa85dec29070cb013174d2e0d4bd" gracePeriod=45
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.947413    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.957428    4872 generic.go:356] "Generic (PLEG): container finished" podID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerID="80f7526d9c96fbc3e2623a532e163de9d03949b43fb5eb8b3d6df57c0775fa70" exitCode=0
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.957511    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" event={"ID":"9583caee-9c57-4cd6-9e29-2765df13ec42","Type":"ContainerDied","Data":"80f7526d9c96fbc3e2623a532e163de9d03949b43fb5eb8b3d6df57c0775fa70"}
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.959920    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" event={"ID":"bbc01fd7-8b47-435a-93aa-91301624879b","Type":"ContainerDied","Data":"5056f5e519d4af34b9253656d87b518c7390fa85dec29070cb013174d2e0d4bd"}
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.960060    4872 generic.go:356] "Generic (PLEG): container finished" podID="bbc01fd7-8b47-435a-93aa-91301624879b" containerID="5056f5e519d4af34b9253656d87b518c7390fa85dec29070cb013174d2e0d4bd" exitCode=0
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.961692    4872 generic.go:356] "Generic (PLEG): container finished" podID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerID="08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803" exitCode=0
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.961862    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" event={"ID":"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5","Type":"ContainerDied","Data":"08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803"}
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.961887    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" event={"ID":"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5","Type":"ContainerDied","Data":"ee7f65eb5dda4a7f95257b930d62d71ced4e7f52f567e6fb184ca8aaf37cb059"}
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.961908    4872 scope.go:122] "RemoveContainer" containerID="08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803"
Nov 01 21:15:52.967244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:52.962055    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"
Nov 01 21:15:53.008579 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.008555    4872 scope.go:122] "RemoveContainer" containerID="08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803"
Nov 01 21:15:53.009077 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:53.009049    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803\": container with ID starting with 08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803 not found: ID does not exist" containerID="08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803"
Nov 01 21:15:53.009189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.009087    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803"} err="failed to get container status \"08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803\": rpc error: code = NotFound desc = could not find container \"08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803\": container with ID starting with 08bb908cc968bc2c82ffba8c3013fcedbbd43f11676dbaf730dce1ec5fb4a803 not found: ID does not exist"
Nov 01 21:15:53.015723 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.015622    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 21:15:53.026028 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.025829    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 21:15:53.081949 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.081916    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm\") pod \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") "
Nov 01 21:15:53.082134 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082084    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home\") pod \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") "
Nov 01 21:15:53.082223 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082204    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs\") pod \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") "
Nov 01 21:15:53.082288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082256    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir\") pod \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") "
Nov 01 21:15:53.082428 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082288    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt\" (UniqueName: \"kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt\") pod \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") "
Nov 01 21:15:53.082428 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082340    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache\") pod \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\" (UID: \"4933e452-e3a8-4ba6-9bfa-b4d07c2acba5\") "
Nov 01 21:15:53.082553 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082448    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home" pod "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" (UID: "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.082636 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082607    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir" pod "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" (UID: "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.082980 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082825    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache" pod "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" (UID: "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.082980 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082938    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.082980 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082950    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.082980 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.082959    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.085690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.085195    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt" pod "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" (UID: "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5"). InnerVolumeSpecName "kube-api-access-j9gjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:53.085690 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.085299    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm" pod "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" (UID: "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.086519 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.086494    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs" pod "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" (UID: "4933e452-e3a8-4ba6-9bfa-b4d07c2acba5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:53.184241 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184205    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm\") pod \"bbc01fd7-8b47-435a-93aa-91301624879b\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") "
Nov 01 21:15:53.184241 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184250    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm\") pod \"9583caee-9c57-4cd6-9e29-2765df13ec42\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184291    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir\") pod \"9583caee-9c57-4cd6-9e29-2765df13ec42\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184347    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz\" (UniqueName: \"kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz\") pod \"9583caee-9c57-4cd6-9e29-2765df13ec42\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184413    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache\") pod \"bbc01fd7-8b47-435a-93aa-91301624879b\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184443    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home\") pod \"bbc01fd7-8b47-435a-93aa-91301624879b\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184467    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs\" (UniqueName: \"kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs\") pod \"9583caee-9c57-4cd6-9e29-2765df13ec42\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184497    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6\" (UniqueName: \"kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6\") pod \"bbc01fd7-8b47-435a-93aa-91301624879b\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") "
Nov 01 21:15:53.184683 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184531    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir\") pod \"bbc01fd7-8b47-435a-93aa-91301624879b\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") "
Nov 01 21:15:53.185374 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184697    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs\") pod \"bbc01fd7-8b47-435a-93aa-91301624879b\" (UID: \"bbc01fd7-8b47-435a-93aa-91301624879b\") "
Nov 01 21:15:53.185374 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184729    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache\") pod \"9583caee-9c57-4cd6-9e29-2765df13ec42\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") "
Nov 01 21:15:53.185374 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.184772    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home\") pod \"9583caee-9c57-4cd6-9e29-2765df13ec42\" (UID: \"9583caee-9c57-4cd6-9e29-2765df13ec42\") "
Nov 01 21:15:53.185374 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.185306    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.185374 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.185318    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home" pod "bbc01fd7-8b47-435a-93aa-91301624879b" (UID: "bbc01fd7-8b47-435a-93aa-91301624879b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.185374 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.185328    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.185691 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.185396    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9gjt\" (UniqueName: \"kubernetes.io/projected/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5-kube-api-access-j9gjt\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.185691 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.185565    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir" pod "bbc01fd7-8b47-435a-93aa-91301624879b" (UID: "bbc01fd7-8b47-435a-93aa-91301624879b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.185691 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.185576    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home" pod "9583caee-9c57-4cd6-9e29-2765df13ec42" (UID: "9583caee-9c57-4cd6-9e29-2765df13ec42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.186387 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.186089    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache" pod "9583caee-9c57-4cd6-9e29-2765df13ec42" (UID: "9583caee-9c57-4cd6-9e29-2765df13ec42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.186554 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.186521    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache" pod "bbc01fd7-8b47-435a-93aa-91301624879b" (UID: "bbc01fd7-8b47-435a-93aa-91301624879b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.186994 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.186970    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir" pod "9583caee-9c57-4cd6-9e29-2765df13ec42" (UID: "9583caee-9c57-4cd6-9e29-2765df13ec42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.188069 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.188029    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs" pod "9583caee-9c57-4cd6-9e29-2765df13ec42" (UID: "9583caee-9c57-4cd6-9e29-2765df13ec42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:53.188555 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.188533    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm" pod "9583caee-9c57-4cd6-9e29-2765df13ec42" (UID: "9583caee-9c57-4cd6-9e29-2765df13ec42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.189577 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.189526    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm" pod "bbc01fd7-8b47-435a-93aa-91301624879b" (UID: "bbc01fd7-8b47-435a-93aa-91301624879b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:53.189704 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.189638    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz" pod "9583caee-9c57-4cd6-9e29-2765df13ec42" (UID: "9583caee-9c57-4cd6-9e29-2765df13ec42"). InnerVolumeSpecName "kube-api-access-j4mxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:53.192027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.190846    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs" pod "bbc01fd7-8b47-435a-93aa-91301624879b" (UID: "bbc01fd7-8b47-435a-93aa-91301624879b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:53.192027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.191214    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6" pod "bbc01fd7-8b47-435a-93aa-91301624879b" (UID: "bbc01fd7-8b47-435a-93aa-91301624879b"). InnerVolumeSpecName "kube-api-access-f9mk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:53.286376 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286344    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286376 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286374    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286387    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9583caee-9c57-4cd6-9e29-2765df13ec42-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286400    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9mk6\" (UniqueName: \"kubernetes.io/projected/bbc01fd7-8b47-435a-93aa-91301624879b-kube-api-access-f9mk6\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286413    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286424    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc01fd7-8b47-435a-93aa-91301624879b-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286446    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286457    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286468    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc01fd7-8b47-435a-93aa-91301624879b-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286478    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286491    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9583caee-9c57-4cd6-9e29-2765df13ec42-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.286598 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.286503    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4mxz\" (UniqueName: \"kubernetes.io/projected/9583caee-9c57-4cd6-9e29-2765df13ec42-kube-api-access-j4mxz\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:53.366710 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:15:53.366666    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:15:53.473289 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.473178    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"]
Nov 01 21:15:53.486225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.486197    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8"]
Nov 01 21:15:53.869803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.869708    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/distinct-2-w0-208f2b-kserve-864ffb6494-s8zk8" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:15:53.933798 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.933755    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:15:53.968296 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.968265    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"
Nov 01 21:15:53.968470 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.968264    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx" event={"ID":"9583caee-9c57-4cd6-9e29-2765df13ec42","Type":"ContainerDied","Data":"6aec59e7673116741d3bc7a2348f0a33754aa68f1d311d901afe59ad179ea7c3"}
Nov 01 21:15:53.968470 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.968395    4872 scope.go:122] "RemoveContainer" containerID="80f7526d9c96fbc3e2623a532e163de9d03949b43fb5eb8b3d6df57c0775fa70"
Nov 01 21:15:53.969751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.969685    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"
Nov 01 21:15:53.969751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.969701    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx" event={"ID":"bbc01fd7-8b47-435a-93aa-91301624879b","Type":"ContainerDied","Data":"96891ceeeaa8c50a7f5acf549d6f2ea0252eb3070f2cdeb0b9014c365c10ca3b"}
Nov 01 21:15:53.977556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:53.977537    4872 scope.go:122] "RemoveContainer" containerID="5056f5e519d4af34b9253656d87b518c7390fa85dec29070cb013174d2e0d4bd"
Nov 01 21:15:54.134272 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:54.134194    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"]
Nov 01 21:15:54.139131 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:54.139106    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/premium-w0-208f2b-kserve-79b4c4d994-ftfxx"]
Nov 01 21:15:54.250928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:54.250895    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"]
Nov 01 21:15:54.254196 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:54.254172    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-208f2b/distinct-w0-208f2b-kserve-5d4ddbf657-ghzlx"]
Nov 01 21:15:55.009562 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:55.009530    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"]
Nov 01 21:15:55.010032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:55.009765    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy" containerID="cri-o://c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455" gracePeriod=30
Nov 01 21:15:55.079409 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:55.079372    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" path="/var/lib/kubelet/pods/4933e452-e3a8-4ba6-9bfa-b4d07c2acba5/volumes"
Nov 01 21:15:55.079977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:55.079955    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" path="/var/lib/kubelet/pods/9583caee-9c57-4cd6-9e29-2765df13ec42/volumes"
Nov 01 21:15:55.080442 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:15:55.080423    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" path="/var/lib/kubelet/pods/bbc01fd7-8b47-435a-93aa-91301624879b/volumes"
Nov 01 21:16:00.282418 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.282348    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 21:16:00.362048 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362019    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362054    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362076    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362099    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362130    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362150    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362216    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362350    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362452    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs\") pod \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\" (UID: \"bcc56ebb-cc6f-467f-b6c5-46df5273f811\") "
Nov 01 21:16:00.362596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362511    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:00.362596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362528    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:00.362839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362768    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:00.362839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362805    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:00.362839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362821    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:00.363012 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.362984    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.363070 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.363014    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.363070 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.363028    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.363070 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.363044    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.363070 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.363057    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.364387 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.364361    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:16:00.364511 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.364486    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:00.364560 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.364521    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "kube-api-access-wdlwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:00.364560 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.364530    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy" pod "bcc56ebb-cc6f-467f-b6c5-46df5273f811" (UID: "bcc56ebb-cc6f-467f-b6c5-46df5273f811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:00.464194 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.464162    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.464194 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.464192    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdlwn\" (UniqueName: \"kubernetes.io/projected/bcc56ebb-cc6f-467f-b6c5-46df5273f811-kube-api-access-wdlwn\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.464382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.464203    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.464382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:00.464213    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc56ebb-cc6f-467f-b6c5-46df5273f811-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:01.017339 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.017300    4872 generic.go:356] "Generic (PLEG): container finished" podID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerID="c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455" exitCode=0
Nov 01 21:16:01.017518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.017352    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"
Nov 01 21:16:01.017518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.017381    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" event={"ID":"bcc56ebb-cc6f-467f-b6c5-46df5273f811","Type":"ContainerDied","Data":"c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455"}
Nov 01 21:16:01.017518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.017439    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq" event={"ID":"bcc56ebb-cc6f-467f-b6c5-46df5273f811","Type":"ContainerDied","Data":"a77ceffeb117519c998434069befe45c59e67ca5c3eb51f2d93a10d0da0b44ff"}
Nov 01 21:16:01.017518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.017470    4872 scope.go:122] "RemoveContainer" containerID="c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455"
Nov 01 21:16:01.027252 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.027228    4872 scope.go:122] "RemoveContainer" containerID="90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"
Nov 01 21:16:01.036248 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.036218    4872 scope.go:122] "RemoveContainer" containerID="c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455"
Nov 01 21:16:01.036504 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:01.036481    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455\": container with ID starting with c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455 not found: ID does not exist" containerID="c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455"
Nov 01 21:16:01.036589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.036517    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455"} err="failed to get container status \"c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455\": rpc error: code = NotFound desc = could not find container \"c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455\": container with ID starting with c05ef4e94e68ae7594e3d02396de44ff86851a5cf49d3f601735c0e092be3455 not found: ID does not exist"
Nov 01 21:16:01.036589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.036541    4872 scope.go:122] "RemoveContainer" containerID="90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"
Nov 01 21:16:01.036818 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:01.036800    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a\": container with ID starting with 90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a not found: ID does not exist" containerID="90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"
Nov 01 21:16:01.036866 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.036826    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a"} err="failed to get container status \"90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a\": rpc error: code = NotFound desc = could not find container \"90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a\": container with ID starting with 90e80cd3fb688758b4b399ea18c561d960879a88b61a0603b0b62ad5c27b7b0a not found: ID does not exist"
Nov 01 21:16:01.184580 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.184546    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"]
Nov 01 21:16:01.235169 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:01.235136    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24f923-openshift-default-645f69b989-dhxtq"]
Nov 01 21:16:03.080030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:03.079992    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" path="/var/lib/kubelet/pods/bcc56ebb-cc6f-467f-b6c5-46df5273f811/volumes"
Nov 01 21:16:04.439034 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.438992    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main" containerID="cri-o://56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab" gracePeriod=45
Nov 01 21:16:04.488562 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.488522    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main" containerID="cri-o://4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2" gracePeriod=45
Nov 01 21:16:04.760021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.759987    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 21:16:04.780923 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.780901    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 21:16:04.811579 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811547    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir\") pod \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") "
Nov 01 21:16:04.811751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811585    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs\" (UniqueName: \"kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs\") pod \"a11224b1-812a-4361-8292-f4d8e2b44095\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") "
Nov 01 21:16:04.811751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811649    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home\") pod \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") "
Nov 01 21:16:04.811751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811704    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7\" (UniqueName: \"kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7\") pod \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") "
Nov 01 21:16:04.811913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811761    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl\" (UniqueName: \"kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl\") pod \"a11224b1-812a-4361-8292-f4d8e2b44095\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") "
Nov 01 21:16:04.811913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811824    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home\") pod \"a11224b1-812a-4361-8292-f4d8e2b44095\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") "
Nov 01 21:16:04.811913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811851    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm\") pod \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") "
Nov 01 21:16:04.811913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811891    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir" pod "97907b4a-9475-4f3f-a0ad-cf6501ee24e0" (UID: "97907b4a-9475-4f3f-a0ad-cf6501ee24e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.811913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811900    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache\") pod \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") "
Nov 01 21:16:04.812419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.811929    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache\") pod \"a11224b1-812a-4361-8292-f4d8e2b44095\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") "
Nov 01 21:16:04.812419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.812000    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir\") pod \"a11224b1-812a-4361-8292-f4d8e2b44095\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") "
Nov 01 21:16:04.812419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.812049    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm\") pod \"a11224b1-812a-4361-8292-f4d8e2b44095\" (UID: \"a11224b1-812a-4361-8292-f4d8e2b44095\") "
Nov 01 21:16:04.812419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.812101    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home" pod "a11224b1-812a-4361-8292-f4d8e2b44095" (UID: "a11224b1-812a-4361-8292-f4d8e2b44095"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.812419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.812122    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs\") pod \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\" (UID: \"97907b4a-9475-4f3f-a0ad-cf6501ee24e0\") "
Nov 01 21:16:04.812695 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.812560    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.812695 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.812580    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.813125 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.813071    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir" pod "a11224b1-812a-4361-8292-f4d8e2b44095" (UID: "a11224b1-812a-4361-8292-f4d8e2b44095"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.813125 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.813083    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache" pod "a11224b1-812a-4361-8292-f4d8e2b44095" (UID: "a11224b1-812a-4361-8292-f4d8e2b44095"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.813329 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.813133    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache" pod "97907b4a-9475-4f3f-a0ad-cf6501ee24e0" (UID: "97907b4a-9475-4f3f-a0ad-cf6501ee24e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.813329 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.813237    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home" pod "97907b4a-9475-4f3f-a0ad-cf6501ee24e0" (UID: "97907b4a-9475-4f3f-a0ad-cf6501ee24e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.814446 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.814421    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7" pod "97907b4a-9475-4f3f-a0ad-cf6501ee24e0" (UID: "97907b4a-9475-4f3f-a0ad-cf6501ee24e0"). InnerVolumeSpecName "kube-api-access-nv2g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:04.814691 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.814672    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm" pod "97907b4a-9475-4f3f-a0ad-cf6501ee24e0" (UID: "97907b4a-9475-4f3f-a0ad-cf6501ee24e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.814777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.814713    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs" pod "a11224b1-812a-4361-8292-f4d8e2b44095" (UID: "a11224b1-812a-4361-8292-f4d8e2b44095"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:04.814840 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.814804    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl" pod "a11224b1-812a-4361-8292-f4d8e2b44095" (UID: "a11224b1-812a-4361-8292-f4d8e2b44095"). InnerVolumeSpecName "kube-api-access-fhvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:04.814947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.814931    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs" pod "97907b4a-9475-4f3f-a0ad-cf6501ee24e0" (UID: "97907b4a-9475-4f3f-a0ad-cf6501ee24e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:04.815103 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.815085    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm" pod "a11224b1-812a-4361-8292-f4d8e2b44095" (UID: "a11224b1-812a-4361-8292-f4d8e2b44095"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:04.913542 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913506    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913542 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913536    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2g7\" (UniqueName: \"kubernetes.io/projected/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-kube-api-access-nv2g7\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913542 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913546    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvrl\" (UniqueName: \"kubernetes.io/projected/a11224b1-812a-4361-8292-f4d8e2b44095-kube-api-access-fhvrl\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913558    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913569    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913578    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913587    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913595    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11224b1-812a-4361-8292-f4d8e2b44095-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913603    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97907b4a-9475-4f3f-a0ad-cf6501ee24e0-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:04.913874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:04.913610    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11224b1-812a-4361-8292-f4d8e2b44095-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:05.041680 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.041574    4872 generic.go:356] "Generic (PLEG): container finished" podID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerID="56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab" exitCode=0
Nov 01 21:16:05.041680 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.041631    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"
Nov 01 21:16:05.041924 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.041675    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" event={"ID":"97907b4a-9475-4f3f-a0ad-cf6501ee24e0","Type":"ContainerDied","Data":"56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab"}
Nov 01 21:16:05.041924 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.041723    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8" event={"ID":"97907b4a-9475-4f3f-a0ad-cf6501ee24e0","Type":"ContainerDied","Data":"0ca235d71d337d9a169ba4aff2b75c0b12e6511d79825a4c2d9cdbac945a6d66"}
Nov 01 21:16:05.041924 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.041747    4872 scope.go:122] "RemoveContainer" containerID="56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab"
Nov 01 21:16:05.043171 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.043153    4872 generic.go:356] "Generic (PLEG): container finished" podID="a11224b1-812a-4361-8292-f4d8e2b44095" containerID="4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2" exitCode=0
Nov 01 21:16:05.043734 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.043197    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"
Nov 01 21:16:05.043734 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.043246    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" event={"ID":"a11224b1-812a-4361-8292-f4d8e2b44095","Type":"ContainerDied","Data":"4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2"}
Nov 01 21:16:05.043734 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.043292    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j" event={"ID":"a11224b1-812a-4361-8292-f4d8e2b44095","Type":"ContainerDied","Data":"5106f3295218030aa452eb1e6802c8db08b122c92815ad111438fc9c32239c27"}
Nov 01 21:16:05.052381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.052248    4872 scope.go:122] "RemoveContainer" containerID="56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab"
Nov 01 21:16:05.052669 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:05.052630    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab\": container with ID starting with 56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab not found: ID does not exist" containerID="56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab"
Nov 01 21:16:05.052772 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.052677    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab"} err="failed to get container status \"56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab\": rpc error: code = NotFound desc = could not find container \"56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab\": container with ID starting with 56ccdd4af9774ec2cf692331754e7be531324917908e4c83b48478f0c117fcab not found: ID does not exist"
Nov 01 21:16:05.052772 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.052702    4872 scope.go:122] "RemoveContainer" containerID="4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2"
Nov 01 21:16:05.063001 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.062985    4872 scope.go:122] "RemoveContainer" containerID="4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2"
Nov 01 21:16:05.063308 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:05.063285    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2\": container with ID starting with 4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2 not found: ID does not exist" containerID="4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2"
Nov 01 21:16:05.063362 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.063320    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2"} err="failed to get container status \"4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2\": rpc error: code = NotFound desc = could not find container \"4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2\": container with ID starting with 4f4a0736527eec940b16bbc4d27828327c334b202d3a6bb606c1cd2d7e5563d2 not found: ID does not exist"
Nov 01 21:16:05.208763 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.208728    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"]
Nov 01 21:16:05.210954 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.210935    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/simulator-w4-1a1e9e-kserve-79549cb6b9-qwk2j"]
Nov 01 21:16:05.320541 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.320457    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"]
Nov 01 21:16:05.326697 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:05.326668    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-2-w4-1a1e9e-kserve-94984776d-w9gj8"]
Nov 01 21:16:07.079073 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:07.079038    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" path="/var/lib/kubelet/pods/97907b4a-9475-4f3f-a0ad-cf6501ee24e0/volumes"
Nov 01 21:16:07.079473 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:07.079459    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" path="/var/lib/kubelet/pods/a11224b1-812a-4361-8292-f4d8e2b44095/volumes"
Nov 01 21:16:12.303278 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:12.303241    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"]
Nov 01 21:16:23.378768 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:23.378711    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:16:24.157549 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:24.157510    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"]
Nov 01 21:16:24.208326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:24.208292    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"]
Nov 01 21:16:25.393388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:25.393307    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"]
Nov 01 21:16:25.393886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:25.393536    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy" containerID="cri-o://28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae" gracePeriod=30
Nov 01 21:16:27.324885 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.324845    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main" containerID="cri-o://0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02" gracePeriod=45
Nov 01 21:16:27.586529 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.586465    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 21:16:27.678805 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.678769    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache\") pod \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") "
Nov 01 21:16:27.678991 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.678819    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm\") pod \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") "
Nov 01 21:16:27.678991 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.678849    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home\") pod \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") "
Nov 01 21:16:27.678991 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.678915    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir\") pod \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") "
Nov 01 21:16:27.678991 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.678963    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs\") pod \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") "
Nov 01 21:16:27.679234 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.679074    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr\" (UniqueName: \"kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr\") pod \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\" (UID: \"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7\") "
Nov 01 21:16:27.679806 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.679383    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache" pod "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" (UID: "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:27.679806 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.679565    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home" pod "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" (UID: "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:27.679941 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.679872    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir" pod "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" (UID: "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:27.681996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.681969    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs" pod "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" (UID: "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:27.681996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.681984    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr" pod "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" (UID: "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7"). InnerVolumeSpecName "kube-api-access-m8czr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:27.682145 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.682018    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm" pod "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" (UID: "22847bb6-c2c4-475e-bfc5-f1d76ffa72f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:27.780709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.780668    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:27.780709 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.780706    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:27.780918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.780719    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:27.780918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.780756    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:27.780918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.780769    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:27.780918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.780781    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8czr\" (UniqueName: \"kubernetes.io/projected/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7-kube-api-access-m8czr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:27.810144 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.810112    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"]
Nov 01 21:16:27.810367 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:27.810334    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" podUID="4dbcc058-f89d-47f3-b56d-18881e425ca6" containerName="maas-api" containerID="cri-o://7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b" gracePeriod=30
Nov 01 21:16:27.826389 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:27.826323    4872 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbcc058_f89d_47f3_b56d_18881e425ca6.slice/crio-conmon-7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 21:16:28.056087 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.056060    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 21:16:28.084489 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.084458    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr\" (UniqueName: \"kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr\") pod \"4dbcc058-f89d-47f3-b56d-18881e425ca6\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") "
Nov 01 21:16:28.084667 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.084570    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca\") pod \"4dbcc058-f89d-47f3-b56d-18881e425ca6\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") "
Nov 01 21:16:28.084667 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.084642    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs\") pod \"4dbcc058-f89d-47f3-b56d-18881e425ca6\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") "
Nov 01 21:16:28.084766 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.084743    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\") pod \"4dbcc058-f89d-47f3-b56d-18881e425ca6\" (UID: \"4dbcc058-f89d-47f3-b56d-18881e425ca6\") "
Nov 01 21:16:28.085000 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.084978    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca" pod "4dbcc058-f89d-47f3-b56d-18881e425ca6" (UID: "4dbcc058-f89d-47f3-b56d-18881e425ca6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:28.085150 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.085129    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbcc058-f89d-47f3-b56d-18881e425ca6-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:28.086604 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.086573    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs" pod "4dbcc058-f89d-47f3-b56d-18881e425ca6" (UID: "4dbcc058-f89d-47f3-b56d-18881e425ca6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:28.087063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.087039    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr" pod "4dbcc058-f89d-47f3-b56d-18881e425ca6" (UID: "4dbcc058-f89d-47f3-b56d-18881e425ca6"). InnerVolumeSpecName "kube-api-access-gcrwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:28.087145 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.087103    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls" pod "4dbcc058-f89d-47f3-b56d-18881e425ca6" (UID: "4dbcc058-f89d-47f3-b56d-18881e425ca6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:28.182047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.181568    4872 generic.go:356] "Generic (PLEG): container finished" podID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerID="0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02" exitCode=0
Nov 01 21:16:28.182047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.181624    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"
Nov 01 21:16:28.182047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.181832    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" event={"ID":"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7","Type":"ContainerDied","Data":"0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02"}
Nov 01 21:16:28.182047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.181883    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh" event={"ID":"22847bb6-c2c4-475e-bfc5-f1d76ffa72f7","Type":"ContainerDied","Data":"b666c308a32354edb9ce7d229e3ade8ae7d349ab95863a6ea690abd9779703c3"}
Nov 01 21:16:28.182047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.181911    4872 scope.go:122] "RemoveContainer" containerID="0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02"
Nov 01 21:16:28.187944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.187900    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:28.187944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.187930    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrwr\" (UniqueName: \"kubernetes.io/projected/4dbcc058-f89d-47f3-b56d-18881e425ca6-kube-api-access-gcrwr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:28.188189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.187950    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dbcc058-f89d-47f3-b56d-18881e425ca6-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:28.204933 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.190798    4872 generic.go:356] "Generic (PLEG): container finished" podID="4dbcc058-f89d-47f3-b56d-18881e425ca6" containerID="7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b" exitCode=0
Nov 01 21:16:28.204933 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.190846    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" event={"ID":"4dbcc058-f89d-47f3-b56d-18881e425ca6","Type":"ContainerDied","Data":"7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b"}
Nov 01 21:16:28.204933 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.190899    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh" event={"ID":"4dbcc058-f89d-47f3-b56d-18881e425ca6","Type":"ContainerDied","Data":"372b31f66e6daacda75537b7a8f2941c2737de090879ded5a766bf0a1b02d7ad"}
Nov 01 21:16:28.204933 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.190985    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"
Nov 01 21:16:28.205246 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.205208    4872 scope.go:122] "RemoveContainer" containerID="0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02"
Nov 01 21:16:28.206289 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:28.206060    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02\": container with ID starting with 0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02 not found: ID does not exist" containerID="0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02"
Nov 01 21:16:28.206289 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.206127    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02"} err="failed to get container status \"0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02\": rpc error: code = NotFound desc = could not find container \"0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02\": container with ID starting with 0b473760d04d8b98fea723234cff1a29459e0d5b4c0b28ed90756c075379de02 not found: ID does not exist"
Nov 01 21:16:28.206289 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.206154    4872 scope.go:122] "RemoveContainer" containerID="7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b"
Nov 01 21:16:28.233631 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.233603    4872 scope.go:122] "RemoveContainer" containerID="7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b"
Nov 01 21:16:28.234525 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:28.234499    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b\": container with ID starting with 7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b not found: ID does not exist" containerID="7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b"
Nov 01 21:16:28.234636 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.234533    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b"} err="failed to get container status \"7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b\": rpc error: code = NotFound desc = could not find container \"7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b\": container with ID starting with 7218dcdee4d87f416076e1511efbf14e394bb73654101b21920fc07a8c2d052b not found: ID does not exist"
Nov 01 21:16:28.347210 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.347175    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"]
Nov 01 21:16:28.353167 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.353112    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/simulator-w1-db0157-kserve-7874796795-gdjjh"]
Nov 01 21:16:28.463554 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.463472    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"]
Nov 01 21:16:28.469302 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:28.469272    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-208f2b-54c76df985-jd6hh"]
Nov 01 21:16:29.078483 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:29.078448    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" path="/var/lib/kubelet/pods/22847bb6-c2c4-475e-bfc5-f1d76ffa72f7/volumes"
Nov 01 21:16:29.078913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:29.078899    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbcc058-f89d-47f3-b56d-18881e425ca6" path="/var/lib/kubelet/pods/4dbcc058-f89d-47f3-b56d-18881e425ca6/volumes"
Nov 01 21:16:30.658612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.658543    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 21:16:30.711797 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.711758    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.711996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.711816    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.711996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.711911    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.711996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.711940    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.711996 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.711981    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.712214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712069    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.712214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712105    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.712214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712144    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.712214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712182    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert\") pod \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\" (UID: \"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7\") "
Nov 01 21:16:30.712214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712191    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:30.712454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712236    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:30.712454 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712257    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:30.712556 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712545    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.712610 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712564    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.712610 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712578    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.712748 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712634    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:30.712748 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.712680    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:30.714273 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.714245    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:16:30.714396 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.714271    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:30.714396 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.714296    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "kube-api-access-dxwk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:30.714529 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.714514    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy" pod "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" (UID: "e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:30.813361 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.813324    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.813361 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.813355    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwk2\" (UniqueName: \"kubernetes.io/projected/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-kube-api-access-dxwk2\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.813361 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.813365    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.813609 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.813374    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.813609 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.813385    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:30.813609 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:30.813394    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:31.210777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.210742    4872 generic.go:356] "Generic (PLEG): container finished" podID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerID="28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae" exitCode=0
Nov 01 21:16:31.210926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.210796    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"
Nov 01 21:16:31.210926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.210823    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" event={"ID":"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7","Type":"ContainerDied","Data":"28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae"}
Nov 01 21:16:31.210926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.210879    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7" event={"ID":"e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7","Type":"ContainerDied","Data":"2a16a0f1a996ebc443701ba1694742129d75661d1665a7690cc500a8a33703fa"}
Nov 01 21:16:31.210926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.210901    4872 scope.go:122] "RemoveContainer" containerID="28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae"
Nov 01 21:16:31.220595 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.220575    4872 scope.go:122] "RemoveContainer" containerID="20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"
Nov 01 21:16:31.229068 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.229042    4872 scope.go:122] "RemoveContainer" containerID="28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae"
Nov 01 21:16:31.229390 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:31.229367    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae\": container with ID starting with 28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae not found: ID does not exist" containerID="28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae"
Nov 01 21:16:31.229474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.229399    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae"} err="failed to get container status \"28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae\": rpc error: code = NotFound desc = could not find container \"28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae\": container with ID starting with 28c8ba258a3565c0683c3c29d13cca19f771737b1c0e2c8bfb54bcdc37758fae not found: ID does not exist"
Nov 01 21:16:31.229474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.229417    4872 scope.go:122] "RemoveContainer" containerID="20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"
Nov 01 21:16:31.229680 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:31.229643    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee\": container with ID starting with 20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee not found: ID does not exist" containerID="20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"
Nov 01 21:16:31.229770 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.229682    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee"} err="failed to get container status \"20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee\": rpc error: code = NotFound desc = could not find container \"20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee\": container with ID starting with 20285d9298c689826f18852e83b030844e67a87212371d06304aec575e173eee not found: ID does not exist"
Nov 01 21:16:31.377627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.377534    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"]
Nov 01 21:16:31.428274 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:31.428239    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-w8rd7"]
Nov 01 21:16:33.080361 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:33.080319    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" path="/var/lib/kubelet/pods/e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7/volumes"
Nov 01 21:16:39.177920 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.177845    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main" containerID="cri-o://144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce" gracePeriod=45
Nov 01 21:16:39.234015 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.233953    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main" containerID="cri-o://77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2" gracePeriod=45
Nov 01 21:16:39.497595 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.497564    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 21:16:39.515076 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.515054    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 21:16:39.629759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.629727    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm\") pod \"37c93251-4cd5-43b6-af0a-6d7af6995c50\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") "
Nov 01 21:16:39.629759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.629761    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir\") pod \"37c93251-4cd5-43b6-af0a-6d7af6995c50\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.629808    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs\") pod \"621e908b-85a1-48af-916d-aaafcb7f15b6\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.629834    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache\") pod \"621e908b-85a1-48af-916d-aaafcb7f15b6\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.629938    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir\") pod \"621e908b-85a1-48af-916d-aaafcb7f15b6\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630012    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs\" (UniqueName: \"kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs\") pod \"37c93251-4cd5-43b6-af0a-6d7af6995c50\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630047    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home\") pod \"37c93251-4cd5-43b6-af0a-6d7af6995c50\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630110    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir" pod "37c93251-4cd5-43b6-af0a-6d7af6995c50" (UID: "37c93251-4cd5-43b6-af0a-6d7af6995c50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630072    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8\" (UniqueName: \"kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8\") pod \"37c93251-4cd5-43b6-af0a-6d7af6995c50\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630210    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm\") pod \"621e908b-85a1-48af-916d-aaafcb7f15b6\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630278    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh\" (UniqueName: \"kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh\") pod \"621e908b-85a1-48af-916d-aaafcb7f15b6\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630308    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home\") pod \"621e908b-85a1-48af-916d-aaafcb7f15b6\" (UID: \"621e908b-85a1-48af-916d-aaafcb7f15b6\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630361    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home" pod "37c93251-4cd5-43b6-af0a-6d7af6995c50" (UID: "37c93251-4cd5-43b6-af0a-6d7af6995c50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630372    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir" pod "621e908b-85a1-48af-916d-aaafcb7f15b6" (UID: "621e908b-85a1-48af-916d-aaafcb7f15b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630383    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache\") pod \"37c93251-4cd5-43b6-af0a-6d7af6995c50\" (UID: \"37c93251-4cd5-43b6-af0a-6d7af6995c50\") "
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630563    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home" pod "621e908b-85a1-48af-916d-aaafcb7f15b6" (UID: "621e908b-85a1-48af-916d-aaafcb7f15b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630576    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache" pod "621e908b-85a1-48af-916d-aaafcb7f15b6" (UID: "621e908b-85a1-48af-916d-aaafcb7f15b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630827    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache" pod "37c93251-4cd5-43b6-af0a-6d7af6995c50" (UID: "37c93251-4cd5-43b6-af0a-6d7af6995c50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630878    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630901    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630914    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630928    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.631147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.630945    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.632710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.632634    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm" pod "37c93251-4cd5-43b6-af0a-6d7af6995c50" (UID: "37c93251-4cd5-43b6-af0a-6d7af6995c50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.632710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.632649    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8" pod "37c93251-4cd5-43b6-af0a-6d7af6995c50" (UID: "37c93251-4cd5-43b6-af0a-6d7af6995c50"). InnerVolumeSpecName "kube-api-access-jfcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:39.632900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.632872    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs" pod "37c93251-4cd5-43b6-af0a-6d7af6995c50" (UID: "37c93251-4cd5-43b6-af0a-6d7af6995c50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:39.633072 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.633050    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh" pod "621e908b-85a1-48af-916d-aaafcb7f15b6" (UID: "621e908b-85a1-48af-916d-aaafcb7f15b6"). InnerVolumeSpecName "kube-api-access-zbnxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:39.633821 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.633799    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs" pod "621e908b-85a1-48af-916d-aaafcb7f15b6" (UID: "621e908b-85a1-48af-916d-aaafcb7f15b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:39.634401 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.634386    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm" pod "621e908b-85a1-48af-916d-aaafcb7f15b6" (UID: "621e908b-85a1-48af-916d-aaafcb7f15b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:16:39.732371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732292    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnxh\" (UniqueName: \"kubernetes.io/projected/621e908b-85a1-48af-916d-aaafcb7f15b6-kube-api-access-zbnxh\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.732371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732327    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.732371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732342    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c93251-4cd5-43b6-af0a-6d7af6995c50-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.732371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732350    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621e908b-85a1-48af-916d-aaafcb7f15b6-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.732371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732358    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c93251-4cd5-43b6-af0a-6d7af6995c50-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.732371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732368    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcv8\" (UniqueName: \"kubernetes.io/projected/37c93251-4cd5-43b6-af0a-6d7af6995c50-kube-api-access-jfcv8\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:39.732689 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:39.732378    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621e908b-85a1-48af-916d-aaafcb7f15b6-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:40.265986 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.265952    4872 generic.go:356] "Generic (PLEG): container finished" podID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerID="77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2" exitCode=0
Nov 01 21:16:40.265986 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.265982    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"
Nov 01 21:16:40.266503 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.266031    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" event={"ID":"37c93251-4cd5-43b6-af0a-6d7af6995c50","Type":"ContainerDied","Data":"77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2"}
Nov 01 21:16:40.266503 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.266070    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz" event={"ID":"37c93251-4cd5-43b6-af0a-6d7af6995c50","Type":"ContainerDied","Data":"631daafc6adba3447f9a5a0b9328eecf68b2581e42b0036bf8a2deddd5f94ff9"}
Nov 01 21:16:40.266503 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.266092    4872 scope.go:122] "RemoveContainer" containerID="77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2"
Nov 01 21:16:40.267461 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.267439    4872 generic.go:356] "Generic (PLEG): container finished" podID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerID="144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce" exitCode=0
Nov 01 21:16:40.267534 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.267477    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"
Nov 01 21:16:40.267534 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.267507    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" event={"ID":"621e908b-85a1-48af-916d-aaafcb7f15b6","Type":"ContainerDied","Data":"144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce"}
Nov 01 21:16:40.267610 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.267534    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz" event={"ID":"621e908b-85a1-48af-916d-aaafcb7f15b6","Type":"ContainerDied","Data":"1e0f86cef5da26f3322ae4b15c91520f914d1ebf092e8dbbae7a4ab4308d7400"}
Nov 01 21:16:40.276115 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.276095    4872 scope.go:122] "RemoveContainer" containerID="77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2"
Nov 01 21:16:40.276454 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:40.276430    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2\": container with ID starting with 77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2 not found: ID does not exist" containerID="77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2"
Nov 01 21:16:40.276597 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.276470    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2"} err="failed to get container status \"77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2\": rpc error: code = NotFound desc = could not find container \"77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2\": container with ID starting with 77d97a6d4dc32fc731e85303fe3b609bfee526d903d02ce59a6e36794bffbbd2 not found: ID does not exist"
Nov 01 21:16:40.276597 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.276493    4872 scope.go:122] "RemoveContainer" containerID="144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce"
Nov 01 21:16:40.285527 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.285506    4872 scope.go:122] "RemoveContainer" containerID="144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce"
Nov 01 21:16:40.285842 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:40.285819    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce\": container with ID starting with 144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce not found: ID does not exist" containerID="144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce"
Nov 01 21:16:40.285917 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.285855    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce"} err="failed to get container status \"144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce\": rpc error: code = NotFound desc = could not find container \"144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce\": container with ID starting with 144c48f84361b1ba2ce23940e8c5ec0297bf29233571102d7b0889176e729dce not found: ID does not exist"
Nov 01 21:16:40.431350 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.431317    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"]
Nov 01 21:16:40.482081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.482008    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-944263/simulator-w3-944263-kserve-665686fdcd-hcjdz"]
Nov 01 21:16:40.546048 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.546012    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"]
Nov 01 21:16:40.549205 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:40.549175    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-944263/premium-w3-944263-kserve-7f5b4bff8c-bzklz"]
Nov 01 21:16:41.078022 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:41.077985    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" path="/var/lib/kubelet/pods/37c93251-4cd5-43b6-af0a-6d7af6995c50/volumes"
Nov 01 21:16:41.078425 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:41.078411    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" path="/var/lib/kubelet/pods/621e908b-85a1-48af-916d-aaafcb7f15b6/volumes"
Nov 01 21:16:49.020393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.020352    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"]
Nov 01 21:16:49.020827 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.020618    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" podUID="fff7ea87-900f-4ece-9d58-d18da67fb3a5" containerName="maas-api" containerID="cri-o://994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca" gracePeriod=30
Nov 01 21:16:49.306263 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.306231    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 21:16:49.316860 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.316823    4872 generic.go:356] "Generic (PLEG): container finished" podID="fff7ea87-900f-4ece-9d58-d18da67fb3a5" containerID="994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca" exitCode=0
Nov 01 21:16:49.317032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.316893    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"
Nov 01 21:16:49.317032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.316924    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" event={"ID":"fff7ea87-900f-4ece-9d58-d18da67fb3a5","Type":"ContainerDied","Data":"994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca"}
Nov 01 21:16:49.317032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.316989    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd" event={"ID":"fff7ea87-900f-4ece-9d58-d18da67fb3a5","Type":"ContainerDied","Data":"73b3b63b240990d2b7ecf58d56a0ed0d3f40bdcef5f73463064b770657f7d98d"}
Nov 01 21:16:49.317032 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.317023    4872 scope.go:122] "RemoveContainer" containerID="994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca"
Nov 01 21:16:49.326607 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.326586    4872 scope.go:122] "RemoveContainer" containerID="994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca"
Nov 01 21:16:49.326951 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:49.326923    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca\": container with ID starting with 994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca not found: ID does not exist" containerID="994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca"
Nov 01 21:16:49.327036 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.326958    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca"} err="failed to get container status \"994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca\": rpc error: code = NotFound desc = could not find container \"994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca\": container with ID starting with 994fba76acd38910d5e0c635221eea4b97f75ae1e0321c09b3b2071662b816ca not found: ID does not exist"
Nov 01 21:16:49.334186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.334156    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs\") pod \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") "
Nov 01 21:16:49.334335 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.334205    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\") pod \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") "
Nov 01 21:16:49.334335 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.334241    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca\") pod \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") "
Nov 01 21:16:49.334449 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.334338    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26\" (UniqueName: \"kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26\") pod \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\" (UID: \"fff7ea87-900f-4ece-9d58-d18da67fb3a5\") "
Nov 01 21:16:49.335071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.335022    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca" pod "fff7ea87-900f-4ece-9d58-d18da67fb3a5" (UID: "fff7ea87-900f-4ece-9d58-d18da67fb3a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:49.336539 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.336506    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs" pod "fff7ea87-900f-4ece-9d58-d18da67fb3a5" (UID: "fff7ea87-900f-4ece-9d58-d18da67fb3a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:49.336684 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.336544    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls" pod "fff7ea87-900f-4ece-9d58-d18da67fb3a5" (UID: "fff7ea87-900f-4ece-9d58-d18da67fb3a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:49.336684 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.336542    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26" pod "fff7ea87-900f-4ece-9d58-d18da67fb3a5" (UID: "fff7ea87-900f-4ece-9d58-d18da67fb3a5"). InnerVolumeSpecName "kube-api-access-l9n26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:49.444303 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.435601    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:49.444303 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.435643    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff7ea87-900f-4ece-9d58-d18da67fb3a5-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:49.444303 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.435677    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff7ea87-900f-4ece-9d58-d18da67fb3a5-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:49.444303 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.435693    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9n26\" (UniqueName: \"kubernetes.io/projected/fff7ea87-900f-4ece-9d58-d18da67fb3a5-kube-api-access-l9n26\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:49.738453 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.738419    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"]
Nov 01 21:16:49.740155 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:49.740133    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-db0157-69688596fc-l76xd"]
Nov 01 21:16:50.190287 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.190252    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"]
Nov 01 21:16:50.190788 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.190450    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" podUID="a0b811c2-632f-4dba-ade7-b3e9a8695e92" containerName="maas-api" containerID="cri-o://cc85cfc34dcd92fdfe8a0a31b6506131cd1374abb56d27569fd3c64d4b232636" gracePeriod=30
Nov 01 21:16:50.323584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.323554    4872 generic.go:356] "Generic (PLEG): container finished" podID="a0b811c2-632f-4dba-ade7-b3e9a8695e92" containerID="cc85cfc34dcd92fdfe8a0a31b6506131cd1374abb56d27569fd3c64d4b232636" exitCode=0
Nov 01 21:16:50.323793 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.323633    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" event={"ID":"a0b811c2-632f-4dba-ade7-b3e9a8695e92","Type":"ContainerDied","Data":"cc85cfc34dcd92fdfe8a0a31b6506131cd1374abb56d27569fd3c64d4b232636"}
Nov 01 21:16:50.450453 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.450391    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 21:16:50.550038 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.549994    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\") pod \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") "
Nov 01 21:16:50.550250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.550116    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs\") pod \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") "
Nov 01 21:16:50.550250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.550170    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj\" (UniqueName: \"kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj\") pod \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") "
Nov 01 21:16:50.550250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.550234    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca\") pod \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\" (UID: \"a0b811c2-632f-4dba-ade7-b3e9a8695e92\") "
Nov 01 21:16:50.550728 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.550695    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca" pod "a0b811c2-632f-4dba-ade7-b3e9a8695e92" (UID: "a0b811c2-632f-4dba-ade7-b3e9a8695e92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:50.552360 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.552336    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj" pod "a0b811c2-632f-4dba-ade7-b3e9a8695e92" (UID: "a0b811c2-632f-4dba-ade7-b3e9a8695e92"). InnerVolumeSpecName "kube-api-access-682gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:50.552360 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.552345    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs" pod "a0b811c2-632f-4dba-ade7-b3e9a8695e92" (UID: "a0b811c2-632f-4dba-ade7-b3e9a8695e92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:50.552500 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.552418    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls" pod "a0b811c2-632f-4dba-ade7-b3e9a8695e92" (UID: "a0b811c2-632f-4dba-ade7-b3e9a8695e92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:50.651603 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.651563    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:50.651603 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.651595    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682gj\" (UniqueName: \"kubernetes.io/projected/a0b811c2-632f-4dba-ade7-b3e9a8695e92-kube-api-access-682gj\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:50.651603 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.651605    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b811c2-632f-4dba-ade7-b3e9a8695e92-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:50.651603 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:50.651614    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b811c2-632f-4dba-ade7-b3e9a8695e92-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:51.077139 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:51.077104    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff7ea87-900f-4ece-9d58-d18da67fb3a5" path="/var/lib/kubelet/pods/fff7ea87-900f-4ece-9d58-d18da67fb3a5/volumes"
Nov 01 21:16:51.330508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:51.330425    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh" event={"ID":"a0b811c2-632f-4dba-ade7-b3e9a8695e92","Type":"ContainerDied","Data":"b17c4b06acaf4af11c83ceb9b0c4e6d0f55e95233ac69a9e32b5c66d24ded257"}
Nov 01 21:16:51.330508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:51.330445    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"
Nov 01 21:16:51.330508 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:51.330492    4872 scope.go:122] "RemoveContainer" containerID="cc85cfc34dcd92fdfe8a0a31b6506131cd1374abb56d27569fd3c64d4b232636"
Nov 01 21:16:51.500403 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:51.500372    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"]
Nov 01 21:16:51.551043 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:51.551012    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-944263-78ffcfd578-n9vrh"]
Nov 01 21:16:53.077811 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:53.077778    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b811c2-632f-4dba-ade7-b3e9a8695e92" path="/var/lib/kubelet/pods/a0b811c2-632f-4dba-ade7-b3e9a8695e92/volumes"
Nov 01 21:16:53.367637 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:16:53.367549    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:16:57.830122 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:57.830089    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"]
Nov 01 21:16:57.830744 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:57.830292    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" containerID="cri-o://6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a" gracePeriod=30
Nov 01 21:16:59.303944 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:59.303908    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"]
Nov 01 21:16:59.304419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:59.304152    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" containerID="cri-o://9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d" gracePeriod=30
Nov 01 21:16:59.905264 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:59.905228    4872 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:16:59.905440 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:16:59.905281    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:17:02.720075 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:02.720038    4872 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:17:02.720495 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:02.720104    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:17:03.089376 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.089352    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 21:17:03.178360 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178330    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178380    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178410    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178438    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178481    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178543    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178588    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:03.178777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178623    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178670    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.178777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178712    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs\") pod \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\" (UID: \"c0b2a14b-93c1-42c7-bca9-72feabde7ffd\") "
Nov 01 21:17:03.179055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.178854    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:03.179055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.179030    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:03.179179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.179157    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.179244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.179186    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.179244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.179202    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.179244 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.179159    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:17:03.179577 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.179394    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:03.180877 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.180851    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:03.180969 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.180901    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:17:03.180969 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.180905    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:03.181333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.181314    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx" pod "c0b2a14b-93c1-42c7-bca9-72feabde7ffd" (UID: "c0b2a14b-93c1-42c7-bca9-72feabde7ffd"). InnerVolumeSpecName "kube-api-access-5q7lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:03.280777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.280702    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.280777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.280730    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.280777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.280741    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.280777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.280750    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7lx\" (UniqueName: \"kubernetes.io/projected/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-kube-api-access-5q7lx\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.280777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.280760    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.280777 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.280770    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b2a14b-93c1-42c7-bca9-72feabde7ffd-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:03.397175 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.397140    4872 generic.go:356] "Generic (PLEG): container finished" podID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerID="6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a" exitCode=0
Nov 01 21:17:03.397175 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.397180    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" event={"ID":"c0b2a14b-93c1-42c7-bca9-72feabde7ffd","Type":"ContainerDied","Data":"6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a"}
Nov 01 21:17:03.397422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.397195    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"
Nov 01 21:17:03.397422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.397225    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj" event={"ID":"c0b2a14b-93c1-42c7-bca9-72feabde7ffd","Type":"ContainerDied","Data":"167c215960911896b20681dc0225de722dd9206491180cf2b8962572789a1b2e"}
Nov 01 21:17:03.397422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.397247    4872 scope.go:122] "RemoveContainer" containerID="6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a"
Nov 01 21:17:03.407105 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.407084    4872 scope.go:122] "RemoveContainer" containerID="6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a"
Nov 01 21:17:03.407352 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:17:03.407333    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a\": container with ID starting with 6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a not found: ID does not exist" containerID="6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a"
Nov 01 21:17:03.407427 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.407360    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a"} err="failed to get container status \"6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a\": rpc error: code = NotFound desc = could not find container \"6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a\": container with ID starting with 6abf831f4499673f0ab7cfbe4ffdc9d74594e474d5b19ba8e713b866ad34057a not found: ID does not exist"
Nov 01 21:17:03.563795 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.563714    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"]
Nov 01 21:17:03.568921 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:03.568900    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-784tj"]
Nov 01 21:17:04.504228 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.504190    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79b4778f-ksf48"]
Nov 01 21:17:04.504985 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.504953    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-d79b4778f-ksf48" podUID="054ff4af-b3f9-4652-b079-549bc614afa8" containerName="manager" containerID="cri-o://f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41" gracePeriod=30
Nov 01 21:17:04.568464 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.568403    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 21:17:04.659309 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.659255    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-d79b4778f-ksf48" podUID="054ff4af-b3f9-4652-b079-549bc614afa8" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.30:8081/readyz\": dial tcp 10.132.0.30:8081: connect: connection refused"
Nov 01 21:17:04.702139 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702101    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702196    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702245    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702291    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702338    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702377    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702435    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702463    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:04.702754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702476    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.702754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.702548    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo\") pod \"f3064070-bfb9-4afe-b20c-20127131740a\" (UID: \"f3064070-bfb9-4afe-b20c-20127131740a\") "
Nov 01 21:17:04.703072 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.703045    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:04.703263 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.703096    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:04.703570 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.703097    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.703672 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.703562    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:17:04.703774 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.703753    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:04.705467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.705439    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:04.705467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.705451    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:17:04.705839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.705807    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "kube-api-access-fcf82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:04.705952 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.705911    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token" pod "f3064070-bfb9-4afe-b20c-20127131740a" (UID: "f3064070-bfb9-4afe-b20c-20127131740a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:04.749984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.749956    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 21:17:04.804093 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804060    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert\") pod \"054ff4af-b3f9-4652-b079-549bc614afa8\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") "
Nov 01 21:17:04.804270 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804171    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7\" (UniqueName: \"kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7\") pod \"054ff4af-b3f9-4652-b079-549bc614afa8\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") "
Nov 01 21:17:04.804270 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804237    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs\") pod \"054ff4af-b3f9-4652-b079-549bc614afa8\" (UID: \"054ff4af-b3f9-4652-b079-549bc614afa8\") "
Nov 01 21:17:04.804532 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804515    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3064070-bfb9-4afe-b20c-20127131740a-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804538    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804552    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804566    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804580    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcf82\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-kube-api-access-fcf82\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804596    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3064070-bfb9-4afe-b20c-20127131740a-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804609    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3064070-bfb9-4afe-b20c-20127131740a-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.804833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.804623    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3064070-bfb9-4afe-b20c-20127131740a-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.806232 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.806211    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs" pod "054ff4af-b3f9-4652-b079-549bc614afa8" (UID: "054ff4af-b3f9-4652-b079-549bc614afa8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:17:04.806307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.806226    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert" pod "054ff4af-b3f9-4652-b079-549bc614afa8" (UID: "054ff4af-b3f9-4652-b079-549bc614afa8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:17:04.806307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.806249    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7" pod "054ff4af-b3f9-4652-b079-549bc614afa8" (UID: "054ff4af-b3f9-4652-b079-549bc614afa8"). InnerVolumeSpecName "kube-api-access-d9fb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:04.905966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.905929    4872 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-webhook-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.905966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.905957    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9fb7\" (UniqueName: \"kubernetes.io/projected/054ff4af-b3f9-4652-b079-549bc614afa8-kube-api-access-d9fb7\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:04.905966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:04.905968    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054ff4af-b3f9-4652-b079-549bc614afa8-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:05.078187 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.078153    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" path="/var/lib/kubelet/pods/c0b2a14b-93c1-42c7-bca9-72feabde7ffd/volumes"
Nov 01 21:17:05.409546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.409513    4872 generic.go:356] "Generic (PLEG): container finished" podID="f3064070-bfb9-4afe-b20c-20127131740a" containerID="9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d" exitCode=0
Nov 01 21:17:05.409725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.409563    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"
Nov 01 21:17:05.409725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.409585    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" event={"ID":"f3064070-bfb9-4afe-b20c-20127131740a","Type":"ContainerDied","Data":"9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d"}
Nov 01 21:17:05.409725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.409640    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v" event={"ID":"f3064070-bfb9-4afe-b20c-20127131740a","Type":"ContainerDied","Data":"a89732bb84b28423e4bd338f1af45ab8d34c18b441a974663b1de3f46a768070"}
Nov 01 21:17:05.409725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.409692    4872 scope.go:122] "RemoveContainer" containerID="9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d"
Nov 01 21:17:05.410934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.410909    4872 generic.go:356] "Generic (PLEG): container finished" podID="054ff4af-b3f9-4652-b079-549bc614afa8" containerID="f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41" exitCode=0
Nov 01 21:17:05.411055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.410959    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-ksf48"
Nov 01 21:17:05.411115 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.411077    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-ksf48" event={"ID":"054ff4af-b3f9-4652-b079-549bc614afa8","Type":"ContainerDied","Data":"f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41"}
Nov 01 21:17:05.411160 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.411114    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-ksf48" event={"ID":"054ff4af-b3f9-4652-b079-549bc614afa8","Type":"ContainerDied","Data":"e14b24ca942825fb6e1c9119bf12246f267f0d13fee88b7e258e9d6d3004a32e"}
Nov 01 21:17:05.418592 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.418576    4872 scope.go:122] "RemoveContainer" containerID="9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d"
Nov 01 21:17:05.418858 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:17:05.418839    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d\": container with ID starting with 9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d not found: ID does not exist" containerID="9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d"
Nov 01 21:17:05.418919 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.418864    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d"} err="failed to get container status \"9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d\": rpc error: code = NotFound desc = could not find container \"9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d\": container with ID starting with 9f93aace6e7efcafd320221a049c47d959a57b6b090774ca8024efbb0f6ddb9d not found: ID does not exist"
Nov 01 21:17:05.418919 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.418879    4872 scope.go:122] "RemoveContainer" containerID="f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41"
Nov 01 21:17:05.427548 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.427391    4872 scope.go:122] "RemoveContainer" containerID="f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41"
Nov 01 21:17:05.427649 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:17:05.427632    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41\": container with ID starting with f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41 not found: ID does not exist" containerID="f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41"
Nov 01 21:17:05.427736 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.427668    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41"} err="failed to get container status \"f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41\": rpc error: code = NotFound desc = could not find container \"f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41\": container with ID starting with f1c4d5a357dbf9ee54f772e5638e5e3f57ebd725feeb63692dbacaa3dc665e41 not found: ID does not exist"
Nov 01 21:17:05.574319 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.574287    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79b4778f-ksf48"]
Nov 01 21:17:05.625064 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.624988    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d79b4778f-ksf48"]
Nov 01 21:17:05.687929 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.687892    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"]
Nov 01 21:17:05.691351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:05.691324    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-np65v"]
Nov 01 21:17:06.451673 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.451626    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d79b4778f-wf9r9"]
Nov 01 21:17:06.452030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452016    4872 cpu_manager.go:414] "containerMap: removing container" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.452071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452031    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.452071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452041    4872 cpu_manager.go:414] "containerMap: removing container" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.452071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452046    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.452071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452062    4872 cpu_manager.go:414] "containerMap: removing container" podUID="a0b811c2-632f-4dba-ade7-b3e9a8695e92" containerName="maas-api"
Nov 01 21:17:06.452071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452068    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b811c2-632f-4dba-ade7-b3e9a8695e92" containerName="maas-api"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452076    4872 cpu_manager.go:414] "containerMap: removing container" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452082    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452091    4872 cpu_manager.go:414] "containerMap: removing container" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452096    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452103    4872 cpu_manager.go:414] "containerMap: removing container" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452108    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452115    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452120    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452125    4872 cpu_manager.go:414] "containerMap: removing container" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452129    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452135    4872 cpu_manager.go:414] "containerMap: removing container" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452139    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452145    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452149    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452156    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452160    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452167    4872 cpu_manager.go:414] "containerMap: removing container" podUID="f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" containerName="maas-api"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452171    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" containerName="maas-api"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452179    4872 cpu_manager.go:414] "containerMap: removing container" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452183    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452189    4872 cpu_manager.go:414] "containerMap: removing container" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452194    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452200    4872 cpu_manager.go:414] "containerMap: removing container" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452204    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452216    4872 cpu_manager.go:414] "containerMap: removing container" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452221    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452230    4872 cpu_manager.go:414] "containerMap: removing container" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main"
Nov 01 21:17:06.452225 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452234    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452242    4872 cpu_manager.go:414] "containerMap: removing container" podUID="66f7599d-77fe-41d6-a9e6-2be4a2515e02" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452246    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f7599d-77fe-41d6-a9e6-2be4a2515e02" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452252    4872 cpu_manager.go:414] "containerMap: removing container" podUID="054ff4af-b3f9-4652-b079-549bc614afa8" containerName="manager"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452257    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054ff4af-b3f9-4652-b079-549bc614afa8" containerName="manager"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452264    4872 cpu_manager.go:414] "containerMap: removing container" podUID="4dbcc058-f89d-47f3-b56d-18881e425ca6" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452269    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbcc058-f89d-47f3-b56d-18881e425ca6" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452279    4872 cpu_manager.go:414] "containerMap: removing container" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452284    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452291    4872 cpu_manager.go:414] "containerMap: removing container" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452294    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452299    4872 cpu_manager.go:414] "containerMap: removing container" podUID="fff7ea87-900f-4ece-9d58-d18da67fb3a5" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452303    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff7ea87-900f-4ece-9d58-d18da67fb3a5" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452310    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452314    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452364    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11224b1-812a-4361-8292-f4d8e2b44095" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452373    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452379    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="621e908b-85a1-48af-916d-aaafcb7f15b6" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452385    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452391    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f7599d-77fe-41d6-a9e6-2be4a2515e02" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452395    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="054ff4af-b3f9-4652-b079-549bc614afa8" containerName="manager"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452402    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452409    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbcc058-f89d-47f3-b56d-18881e425ca6" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452414    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452420    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="22847bb6-c2c4-475e-bfc5-f1d76ffa72f7" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452426    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="4933e452-e3a8-4ba6-9bfa-b4d07c2acba5" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452431    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff7ea87-900f-4ece-9d58-d18da67fb3a5" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452436    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452441    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452446    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452453    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b811c2-632f-4dba-ade7-b3e9a8695e92" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452457    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c93251-4cd5-43b6-af0a-6d7af6995c50" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452462    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452467    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc01fd7-8b47-435a-93aa-91301624879b" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452473    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="9583caee-9c57-4cd6-9e29-2765df13ec42" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452479    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452484    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b2a14b-93c1-42c7-bca9-72feabde7ffd" containerName="istio-proxy"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452489    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3064070-bfb9-4afe-b20c-20127131740a" containerName="istio-proxy"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452494    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5baa56e-8648-45ab-9c9f-cc4b1cae7dc1" containerName="maas-api"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452499    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="97907b4a-9475-4f3f-a0ad-cf6501ee24e0" containerName="main"
Nov 01 21:17:06.453074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.452504    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc56ebb-cc6f-467f-b6c5-46df5273f811" containerName="istio-proxy"
Nov 01 21:17:06.456993 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.456975    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.510421 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.510378    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 21:17:06.523526 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.523506    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.523586 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.523560    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cwd\" (UniqueName: \"kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.523722 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.523708    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.561305 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.561270    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nm6vj\""
Nov 01 21:17:06.561456 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.561363    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 21:17:06.561456 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.561421    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 21:17:06.563128 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.563108    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4778f-wf9r9"]
Nov 01 21:17:06.624729 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.624699    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.625110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.624756    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.625110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.624806    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cwd\" (UniqueName: \"kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.627304 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.627281    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.627362 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.627345    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.683305 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.683275    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cwd\" (UniqueName: \"kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd\") pod \"maas-controller-d79b4778f-wf9r9\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") " pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:06.770075 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:06.769994    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:07.078557 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:07.078483    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054ff4af-b3f9-4652-b079-549bc614afa8" path="/var/lib/kubelet/pods/054ff4af-b3f9-4652-b079-549bc614afa8/volumes"
Nov 01 21:17:07.079008 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:07.078994    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3064070-bfb9-4afe-b20c-20127131740a" path="/var/lib/kubelet/pods/f3064070-bfb9-4afe-b20c-20127131740a/volumes"
Nov 01 21:17:07.143497 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:07.143474    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4778f-wf9r9"]
Nov 01 21:17:07.145967 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:17:07.145937    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8c4118_c87a_4615_9876_1ba044834af2.slice/crio-83cb9cb2014d2adc02ed72605c5b25ed8ffeb20df6e087246c59943090ac0abd WatchSource:0}: Error finding container 83cb9cb2014d2adc02ed72605c5b25ed8ffeb20df6e087246c59943090ac0abd: Status 404 returned error can't find the container with id 83cb9cb2014d2adc02ed72605c5b25ed8ffeb20df6e087246c59943090ac0abd
Nov 01 21:17:07.147195 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:07.147179    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:17:07.424320 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:07.424284    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-wf9r9" event={"ID":"ba8c4118-c87a-4615-9876-1ba044834af2","Type":"ContainerStarted","Data":"83cb9cb2014d2adc02ed72605c5b25ed8ffeb20df6e087246c59943090ac0abd"}
Nov 01 21:17:08.430375 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:08.430341    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-wf9r9" event={"ID":"ba8c4118-c87a-4615-9876-1ba044834af2","Type":"ContainerStarted","Data":"220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54"}
Nov 01 21:17:08.430932 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:08.430433    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:08.547415 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:08.547362    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d79b4778f-wf9r9" podStartSLOduration=1.636332746 podStartE2EDuration="2.54734939s" totalImagesPullingTime="911.016644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:08.5458502 +0000 UTC m=+2785.594960020" watchObservedRunningTime="2026-11-01 21:17:08.54734939 +0000 UTC m=+2785.596459208"
Nov 01 21:17:19.437858 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:19.437817    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:23.364611 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:17:23.364564    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:17:39.762019 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:39.761981    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79b4778f-wf9r9"]
Nov 01 21:17:39.762543 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:39.762176    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-d79b4778f-wf9r9" podUID="ba8c4118-c87a-4615-9876-1ba044834af2" containerName="manager" containerID="cri-o://220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54" gracePeriod=30
Nov 01 21:17:40.016058 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.015995    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:40.094033 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.093999    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert\") pod \"ba8c4118-c87a-4615-9876-1ba044834af2\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") "
Nov 01 21:17:40.094033 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.094039    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs\") pod \"ba8c4118-c87a-4615-9876-1ba044834af2\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") "
Nov 01 21:17:40.094306 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.094116    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd\" (UniqueName: \"kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd\") pod \"ba8c4118-c87a-4615-9876-1ba044834af2\" (UID: \"ba8c4118-c87a-4615-9876-1ba044834af2\") "
Nov 01 21:17:40.096261 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.096228    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs" pod "ba8c4118-c87a-4615-9876-1ba044834af2" (UID: "ba8c4118-c87a-4615-9876-1ba044834af2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:17:40.096261 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.096240    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd" pod "ba8c4118-c87a-4615-9876-1ba044834af2" (UID: "ba8c4118-c87a-4615-9876-1ba044834af2"). InnerVolumeSpecName "kube-api-access-r6cwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:40.096417 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.096246    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert" pod "ba8c4118-c87a-4615-9876-1ba044834af2" (UID: "ba8c4118-c87a-4615-9876-1ba044834af2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:17:40.195936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.195905    4872 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-webhook-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:40.195936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.195932    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8c4118-c87a-4615-9876-1ba044834af2-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:40.195936 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.195943    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cwd\" (UniqueName: \"kubernetes.io/projected/ba8c4118-c87a-4615-9876-1ba044834af2-kube-api-access-r6cwd\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:40.597201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.597162    4872 generic.go:356] "Generic (PLEG): container finished" podID="ba8c4118-c87a-4615-9876-1ba044834af2" containerID="220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54" exitCode=0
Nov 01 21:17:40.597405 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.597233    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-wf9r9"
Nov 01 21:17:40.597405 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.597244    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-wf9r9" event={"ID":"ba8c4118-c87a-4615-9876-1ba044834af2","Type":"ContainerDied","Data":"220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54"}
Nov 01 21:17:40.597405 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.597328    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-wf9r9" event={"ID":"ba8c4118-c87a-4615-9876-1ba044834af2","Type":"ContainerDied","Data":"83cb9cb2014d2adc02ed72605c5b25ed8ffeb20df6e087246c59943090ac0abd"}
Nov 01 21:17:40.597405 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.597350    4872 scope.go:122] "RemoveContainer" containerID="220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54"
Nov 01 21:17:40.606547 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.606523    4872 scope.go:122] "RemoveContainer" containerID="220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54"
Nov 01 21:17:40.606895 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:17:40.606871    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54\": container with ID starting with 220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54 not found: ID does not exist" containerID="220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54"
Nov 01 21:17:40.606973 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.606907    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54"} err="failed to get container status \"220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54\": rpc error: code = NotFound desc = could not find container \"220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54\": container with ID starting with 220cd2ef2f0aec15bccc762f4f11773b112f76edbbc6f4a6a397328b7154af54 not found: ID does not exist"
Nov 01 21:17:40.764463 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.764426    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79b4778f-wf9r9"]
Nov 01 21:17:40.815202 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:40.815167    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d79b4778f-wf9r9"]
Nov 01 21:17:41.078188 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:41.078152    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8c4118-c87a-4615-9876-1ba044834af2" path="/var/lib/kubelet/pods/ba8c4118-c87a-4615-9876-1ba044834af2/volumes"
Nov 01 21:17:42.061808 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.061778    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d79b4778f-bt8rf"]
Nov 01 21:17:42.062201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062163    4872 cpu_manager.go:414] "containerMap: removing container" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy"
Nov 01 21:17:42.062201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062174    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1342e2c8-5e98-48e3-8e34-3ea77dc9c74f" containerName="istio-proxy"
Nov 01 21:17:42.062201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062193    4872 cpu_manager.go:414] "containerMap: removing container" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy"
Nov 01 21:17:42.062201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062199    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062213    4872 cpu_manager.go:414] "containerMap: removing container" podUID="ba8c4118-c87a-4615-9876-1ba044834af2" containerName="manager"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062218    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8c4118-c87a-4615-9876-1ba044834af2" containerName="manager"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062235    4872 cpu_manager.go:414] "containerMap: removing container" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062240    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062246    4872 cpu_manager.go:414] "containerMap: removing container" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062251    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0184315a-2824-4828-a824-a6ed7438150d" containerName="cleanup"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062305    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8c4118-c87a-4615-9876-1ba044834af2" containerName="manager"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062312    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e9d24b-2ccb-4ade-8a6c-9c7fac6c27e7" containerName="istio-proxy"
Nov 01 21:17:42.062334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.062319    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8125de3-d3f2-40c2-a169-9a733fadaf1b" containerName="cleanup"
Nov 01 21:17:42.066757 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.066736    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.170481 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.170450    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 21:17:42.170718 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.170693    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 21:17:42.170912 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.170741    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nm6vj\""
Nov 01 21:17:42.170912 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.170782    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 21:17:42.172635 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.172615    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4778f-bt8rf"]
Nov 01 21:17:42.216175 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.216144    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcx86\" (UniqueName: \"kubernetes.io/projected/c54edc37-7504-492c-8386-5bfef499d4fa-kube-api-access-qcx86\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.216392 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.216379    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54edc37-7504-492c-8386-5bfef499d4fa-webhook-cert\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.216474 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.216414    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54edc37-7504-492c-8386-5bfef499d4fa-metrics-certs\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.317825 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.317727    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54edc37-7504-492c-8386-5bfef499d4fa-webhook-cert\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.317825 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.317782    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54edc37-7504-492c-8386-5bfef499d4fa-metrics-certs\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.318086 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.317853    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcx86\" (UniqueName: \"kubernetes.io/projected/c54edc37-7504-492c-8386-5bfef499d4fa-kube-api-access-qcx86\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.320494 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.320466    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c54edc37-7504-492c-8386-5bfef499d4fa-metrics-certs\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.320612 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.320466    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54edc37-7504-492c-8386-5bfef499d4fa-webhook-cert\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.375185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.375148    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcx86\" (UniqueName: \"kubernetes.io/projected/c54edc37-7504-492c-8386-5bfef499d4fa-kube-api-access-qcx86\") pod \"maas-controller-d79b4778f-bt8rf\" (UID: \"c54edc37-7504-492c-8386-5bfef499d4fa\") " pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.378935 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.378910    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:42.759319 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:42.759296    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4778f-bt8rf"]
Nov 01 21:17:42.762035 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:17:42.762003    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54edc37_7504_492c_8386_5bfef499d4fa.slice/crio-6295b7f91e44a177705dc078685939d97b033432e9e0b5369a87ba287704a5a4 WatchSource:0}: Error finding container 6295b7f91e44a177705dc078685939d97b033432e9e0b5369a87ba287704a5a4: Status 404 returned error can't find the container with id 6295b7f91e44a177705dc078685939d97b033432e9e0b5369a87ba287704a5a4
Nov 01 21:17:43.614633 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:43.614597    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-bt8rf" event={"ID":"c54edc37-7504-492c-8386-5bfef499d4fa","Type":"ContainerStarted","Data":"6295b7f91e44a177705dc078685939d97b033432e9e0b5369a87ba287704a5a4"}
Nov 01 21:17:43.623176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:43.623150    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 21:17:44.621015 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:44.620976    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4778f-bt8rf" event={"ID":"c54edc37-7504-492c-8386-5bfef499d4fa","Type":"ContainerStarted","Data":"d699a9affa8eca3bd9f6271166001413f2e4a6aac2f2a677e82e64bb765a7c8b"}
Nov 01 21:17:44.621419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:44.621105    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:17:44.737461 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:44.737406    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d79b4778f-bt8rf" podStartSLOduration=1.929952516 podStartE2EDuration="2.737393887s" totalImagesPullingTime="807.441371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:44.736904975 +0000 UTC m=+2821.786014793" watchObservedRunningTime="2026-11-01 21:17:44.737393887 +0000 UTC m=+2821.786503705"
Nov 01 21:17:53.365804 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:17:53.365765    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:17:55.627529 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:17:55.627447    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d79b4778f-bt8rf"
Nov 01 21:18:23.365476 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:18:23.365436    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:18:53.365632 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:18:53.365598    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:19:23.365176 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:19:23.365137    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:19:53.365140 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:19:53.365099    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:20:23.365451 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:20:23.365414    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:20:43.608186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:20:43.608156    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:20:43.612206 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:20:43.612175    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:20:43.616295 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:20:43.616268    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:20:43.620911 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:20:43.620892    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:20:53.365112 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:20:53.365074    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:21:23.365052 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:21:23.365013    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:21:53.364967 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:21:53.364924    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:22:23.364287 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:22:23.364252    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:22:53.365112 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:22:53.365071    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:23:23.364595 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:23:23.364556    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:23:53.369148 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:23:53.369103    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:24:23.365286 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:24:23.365225    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:24:53.364615 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:24:53.364579    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:25:16.453823 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.453787    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"]
Nov 01 21:25:16.457421 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.457397    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.514337 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.514302    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"]
Nov 01 21:25:16.528269 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.528238    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.528405 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.528318    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmcd\" (UniqueName: \"kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd\") pod \"payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.561989 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.561966    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7249f4\""
Nov 01 21:25:16.562105 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.562079    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e65bae51-dockercfg-mrpkb\""
Nov 01 21:25:16.629444 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.629411    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.629588 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.629464    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmcd\" (UniqueName: \"kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd\") pod \"payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.630146 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.630125    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.686759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.686728    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmcd\" (UniqueName: \"kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd\") pod \"payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:16.771213 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:16.771146    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:17.144679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:17.144635    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"]
Nov 01 21:25:17.147014 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:25:17.146985    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eccb57f_3641_4004_974d_a803bbbeb19c.slice/crio-c759183c8dc9aa4fe673c9e671e707e9cabdde727535fc215bb4e9ed8e3caee5 WatchSource:0}: Error finding container c759183c8dc9aa4fe673c9e671e707e9cabdde727535fc215bb4e9ed8e3caee5: Status 404 returned error can't find the container with id c759183c8dc9aa4fe673c9e671e707e9cabdde727535fc215bb4e9ed8e3caee5
Nov 01 21:25:17.969506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:17.969474    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" event={"ID":"9eccb57f-3641-4004-974d-a803bbbeb19c","Type":"ContainerStarted","Data":"2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24"}
Nov 01 21:25:17.969939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:17.969523    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" event={"ID":"9eccb57f-3641-4004-974d-a803bbbeb19c","Type":"ContainerStarted","Data":"c759183c8dc9aa4fe673c9e671e707e9cabdde727535fc215bb4e9ed8e3caee5"}
Nov 01 21:25:17.969939 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:17.969582    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:18.085302 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.085254    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" podStartSLOduration=2.0852413 podStartE2EDuration="2.0852413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:18.083627305 +0000 UTC m=+3275.132737124" watchObservedRunningTime="2026-11-01 21:25:18.0852413 +0000 UTC m=+3275.134351118"
Nov 01 21:25:18.760562 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.760526    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"]
Nov 01 21:25:18.764195 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.764178    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.853488 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.853454    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv767\" (UniqueName: \"kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.853686 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.853499    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.853686 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.853517    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.853686 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.853671    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.869722 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.869697    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7249f4-serving-cert\""
Nov 01 21:25:18.871644 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.871624    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"]
Nov 01 21:25:18.955134 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.955103    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.955307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.955172    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv767\" (UniqueName: \"kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.955307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.955212    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.955307 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.955233    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.955307 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:25:18.955252    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-serving-cert: secret "maas-api-e2e-worker-single-main-7249f4-serving-cert" not found
Nov 01 21:25:18.955539 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:25:18.955314    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls podName:7204442b-7399-4f79-a50e-2073fdb73704 nodeName:}" failed. No retries permitted until 2026-11-01 21:25:19.455293043 +0000 UTC m=+3276.504402844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls") pod "maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" (UID: "7204442b-7399-4f79-a50e-2073fdb73704") : secret "maas-api-e2e-worker-single-main-7249f4-serving-cert" not found
Nov 01 21:25:18.955960 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.955936    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:18.957592 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:18.957572    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:19.013578 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:19.013477    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv767\" (UniqueName: \"kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:19.459884 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:19.459854    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:19.462280 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:19.462255    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:19.676951 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:19.676922    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:20.050518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:20.050493    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"]
Nov 01 21:25:20.052378 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:25:20.052350    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7204442b_7399_4f79_a50e_2073fdb73704.slice/crio-92d827d883c1c33040087780124099ade258fb0153b10cee7227af9756541db9 WatchSource:0}: Error finding container 92d827d883c1c33040087780124099ade258fb0153b10cee7227af9756541db9: Status 404 returned error can't find the container with id 92d827d883c1c33040087780124099ade258fb0153b10cee7227af9756541db9
Nov 01 21:25:20.053832 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:20.053813    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:25:20.985763 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:20.985737    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" event={"ID":"7204442b-7399-4f79-a50e-2073fdb73704","Type":"ContainerStarted","Data":"92d827d883c1c33040087780124099ade258fb0153b10cee7227af9756541db9"}
Nov 01 21:25:21.992106 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:21.992074    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" event={"ID":"7204442b-7399-4f79-a50e-2073fdb73704","Type":"ContainerStarted","Data":"ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173"}
Nov 01 21:25:21.992913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:21.992194    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:22.109411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:22.109361    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" podStartSLOduration=3.221785658 podStartE2EDuration="4.1093476s" totalImagesPullingTime="887.561942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:22.107034522 +0000 UTC m=+3279.156144340" watchObservedRunningTime="2026-11-01 21:25:22.1093476 +0000 UTC m=+3279.158457416"
Nov 01 21:25:23.364093 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:25:23.364059    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:25:28.002852 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:28.002773    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:25:28.978184 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:28.978157    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:25:40.867576 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.867542    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"]
Nov 01 21:25:40.872297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.872276    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.928018 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.927987    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"]
Nov 01 21:25:40.974411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.974381    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.974582 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.974451    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.974582 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.974536    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.974582 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.974573    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.974710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.974591    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vzg\" (UniqueName: \"kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.974745 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.974715    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:40.975872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.975853    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7249f4\"/\"openshift-service-ca.crt\""
Nov 01 21:25:40.975927 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.975910    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7249f4\"/\"kube-root-ca.crt\""
Nov 01 21:25:40.975966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.975945    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7249f4\"/\"simulator-single-main-7249f4-kserve-self-signed-certs\""
Nov 01 21:25:40.976352 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:40.976338    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7249f4\"/\"default-dockercfg-cjnq2\""
Nov 01 21:25:41.075773 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.075734    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.075983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.075794    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.075983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.075817    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vzg\" (UniqueName: \"kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.075983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.075866    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.076149 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.076002    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.076149 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.076072    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.077015 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.076986    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.077015 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.077008    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.077217 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.077056    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.079224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.079199    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.079325 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.079247    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.136386 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.136324    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vzg\" (UniqueName: \"kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg\") pod \"simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") " pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.184566 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.184538    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:41.558279 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:41.558254    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"]
Nov 01 21:25:41.560340 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:25:41.560314    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fe11e5b_caae_44bf_902c_24870cfb8b50.slice/crio-e8c10c2feec0d49d96b04ac453660d0363c726cd4251ea742924c56e3595b0c1 WatchSource:0}: Error finding container e8c10c2feec0d49d96b04ac453660d0363c726cd4251ea742924c56e3595b0c1: Status 404 returned error can't find the container with id e8c10c2feec0d49d96b04ac453660d0363c726cd4251ea742924c56e3595b0c1
Nov 01 21:25:42.087200 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:42.087154    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" event={"ID":"7fe11e5b-caae-44bf-902c-24870cfb8b50","Type":"ContainerStarted","Data":"f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362"}
Nov 01 21:25:42.087618 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:42.087204    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" event={"ID":"7fe11e5b-caae-44bf-902c-24870cfb8b50","Type":"ContainerStarted","Data":"e8c10c2feec0d49d96b04ac453660d0363c726cd4251ea742924c56e3595b0c1"}
Nov 01 21:25:42.185708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:42.185676    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:42.213742 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:42.213684    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" podStartSLOduration=2.213647287 podStartE2EDuration="2.213647287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:42.212891061 +0000 UTC m=+3299.262000880" watchObservedRunningTime="2026-11-01 21:25:42.213647287 +0000 UTC m=+3299.262757104"
Nov 01 21:25:43.641241 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:43.641209    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:25:43.645977 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:43.645955    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:25:43.647204 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:43.647181    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:25:43.651611 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:43.651594    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:25:51.185627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:51.185594    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:51.198826 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:51.198797    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:52.144085 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:25:52.144052    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:25:53.365281 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:25:53.365247    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:26:23.365607 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:26:23.365569    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:26:42.087872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:42.087836    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"]
Nov 01 21:26:49.094705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:49.094666    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"]
Nov 01 21:26:53.365985 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:26:53.365941    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:26:54.095291 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:54.095223    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerName="payload-processing" containerID="cri-o://2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24" gracePeriod=25
Nov 01 21:26:54.892492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:54.892461    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"]
Nov 01 21:26:54.892995 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:54.892698    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" podUID="7204442b-7399-4f79-a50e-2073fdb73704" containerName="maas-api" containerID="cri-o://ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173" gracePeriod=30
Nov 01 21:26:56.156859 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.156833    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:26:56.226371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.226277    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\") pod \"7204442b-7399-4f79-a50e-2073fdb73704\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") "
Nov 01 21:26:56.226371 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.226360    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767\" (UniqueName: \"kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767\") pod \"7204442b-7399-4f79-a50e-2073fdb73704\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") "
Nov 01 21:26:56.226626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.226495    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs\") pod \"7204442b-7399-4f79-a50e-2073fdb73704\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") "
Nov 01 21:26:56.226626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.226528    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca\") pod \"7204442b-7399-4f79-a50e-2073fdb73704\" (UID: \"7204442b-7399-4f79-a50e-2073fdb73704\") "
Nov 01 21:26:56.227019 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.226948    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca" pod "7204442b-7399-4f79-a50e-2073fdb73704" (UID: "7204442b-7399-4f79-a50e-2073fdb73704"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:26:56.229095 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.229064    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs" pod "7204442b-7399-4f79-a50e-2073fdb73704" (UID: "7204442b-7399-4f79-a50e-2073fdb73704"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:26:56.229198 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.229092    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767" pod "7204442b-7399-4f79-a50e-2073fdb73704" (UID: "7204442b-7399-4f79-a50e-2073fdb73704"). InnerVolumeSpecName "kube-api-access-nv767". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:26:56.229198 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.229133    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls" pod "7204442b-7399-4f79-a50e-2073fdb73704" (UID: "7204442b-7399-4f79-a50e-2073fdb73704"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:26:56.328063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.328021    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:56.328063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.328060    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7204442b-7399-4f79-a50e-2073fdb73704-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:56.328063 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.328070    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7204442b-7399-4f79-a50e-2073fdb73704-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:56.328314 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.328079    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv767\" (UniqueName: \"kubernetes.io/projected/7204442b-7399-4f79-a50e-2073fdb73704-kube-api-access-nv767\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:56.447229 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.447191    4872 generic.go:356] "Generic (PLEG): container finished" podID="7204442b-7399-4f79-a50e-2073fdb73704" containerID="ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173" exitCode=0
Nov 01 21:26:56.447433 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.447242    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"
Nov 01 21:26:56.447433 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.447276    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" event={"ID":"7204442b-7399-4f79-a50e-2073fdb73704","Type":"ContainerDied","Data":"ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173"}
Nov 01 21:26:56.447433 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.447331    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz" event={"ID":"7204442b-7399-4f79-a50e-2073fdb73704","Type":"ContainerDied","Data":"92d827d883c1c33040087780124099ade258fb0153b10cee7227af9756541db9"}
Nov 01 21:26:56.447433 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.447353    4872 scope.go:122] "RemoveContainer" containerID="ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173"
Nov 01 21:26:56.460956 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.460938    4872 scope.go:122] "RemoveContainer" containerID="ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173"
Nov 01 21:26:56.461280 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:26:56.461260    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173\": container with ID starting with ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173 not found: ID does not exist" containerID="ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173"
Nov 01 21:26:56.461349 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.461295    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173"} err="failed to get container status \"ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173\": rpc error: code = NotFound desc = could not find container \"ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173\": container with ID starting with ebab9f79926242d9d2b2089f592860228cdac33ab8482cd64618a1fe402ee173 not found: ID does not exist"
Nov 01 21:26:56.612459 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.612359    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"]
Nov 01 21:26:56.615943 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:56.615913    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7249f4-69757f9654-zbsrz"]
Nov 01 21:26:57.078741 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.078707    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7204442b-7399-4f79-a50e-2073fdb73704" path="/var/lib/kubelet/pods/7204442b-7399-4f79-a50e-2073fdb73704/volumes"
Nov 01 21:26:57.107964 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.107922    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" podUID="7fe11e5b-caae-44bf-902c-24870cfb8b50" containerName="main" containerID="cri-o://f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362" gracePeriod=45
Nov 01 21:26:57.133573 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.133539    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" podUID="7fe11e5b-caae-44bf-902c-24870cfb8b50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": dial tcp 10.132.0.135:8000: connect: connection refused"
Nov 01 21:26:57.370202 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.370173    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:26:57.438847 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.438806    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg\" (UniqueName: \"kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg\") pod \"7fe11e5b-caae-44bf-902c-24870cfb8b50\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") "
Nov 01 21:26:57.439052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.438859    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir\") pod \"7fe11e5b-caae-44bf-902c-24870cfb8b50\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") "
Nov 01 21:26:57.439052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.438904    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs\") pod \"7fe11e5b-caae-44bf-902c-24870cfb8b50\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") "
Nov 01 21:26:57.439052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.438983    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home\") pod \"7fe11e5b-caae-44bf-902c-24870cfb8b50\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") "
Nov 01 21:26:57.439052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439017    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache\") pod \"7fe11e5b-caae-44bf-902c-24870cfb8b50\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") "
Nov 01 21:26:57.439052 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439041    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm\") pod \"7fe11e5b-caae-44bf-902c-24870cfb8b50\" (UID: \"7fe11e5b-caae-44bf-902c-24870cfb8b50\") "
Nov 01 21:26:57.439322 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439179    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir" pod "7fe11e5b-caae-44bf-902c-24870cfb8b50" (UID: "7fe11e5b-caae-44bf-902c-24870cfb8b50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:26:57.439369 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439315    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home" pod "7fe11e5b-caae-44bf-902c-24870cfb8b50" (UID: "7fe11e5b-caae-44bf-902c-24870cfb8b50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:26:57.439419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439397    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache" pod "7fe11e5b-caae-44bf-902c-24870cfb8b50" (UID: "7fe11e5b-caae-44bf-902c-24870cfb8b50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:26:57.439419 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439413    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:57.439512 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.439430    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:57.441328 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.441298    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm" pod "7fe11e5b-caae-44bf-902c-24870cfb8b50" (UID: "7fe11e5b-caae-44bf-902c-24870cfb8b50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:26:57.441328 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.441320    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg" pod "7fe11e5b-caae-44bf-902c-24870cfb8b50" (UID: "7fe11e5b-caae-44bf-902c-24870cfb8b50"). InnerVolumeSpecName "kube-api-access-55vzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:26:57.441487 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.441399    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs" pod "7fe11e5b-caae-44bf-902c-24870cfb8b50" (UID: "7fe11e5b-caae-44bf-902c-24870cfb8b50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:26:57.455268 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.455234    4872 generic.go:356] "Generic (PLEG): container finished" podID="7fe11e5b-caae-44bf-902c-24870cfb8b50" containerID="f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362" exitCode=0
Nov 01 21:26:57.455404 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.455320    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" event={"ID":"7fe11e5b-caae-44bf-902c-24870cfb8b50","Type":"ContainerDied","Data":"f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362"}
Nov 01 21:26:57.455404 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.455357    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp" event={"ID":"7fe11e5b-caae-44bf-902c-24870cfb8b50","Type":"ContainerDied","Data":"e8c10c2feec0d49d96b04ac453660d0363c726cd4251ea742924c56e3595b0c1"}
Nov 01 21:26:57.455404 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.455383    4872 scope.go:122] "RemoveContainer" containerID="f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362"
Nov 01 21:26:57.455521 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.455487    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"
Nov 01 21:26:57.465863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.465842    4872 scope.go:122] "RemoveContainer" containerID="f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362"
Nov 01 21:26:57.466244 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:26:57.466225    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362\": container with ID starting with f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362 not found: ID does not exist" containerID="f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362"
Nov 01 21:26:57.466302 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.466257    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362"} err="failed to get container status \"f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362\": rpc error: code = NotFound desc = could not find container \"f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362\": container with ID starting with f20e54e3963586921331418e4e55e401c39fb045ec3fb55f21614ff3db8c4362 not found: ID does not exist"
Nov 01 21:26:57.540290 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.540257    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:57.540290 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.540285    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe11e5b-caae-44bf-902c-24870cfb8b50-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:57.540505 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.540302    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vzg\" (UniqueName: \"kubernetes.io/projected/7fe11e5b-caae-44bf-902c-24870cfb8b50-kube-api-access-55vzg\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:57.540505 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.540314    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe11e5b-caae-44bf-902c-24870cfb8b50-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:57.599117 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.599036    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"]
Nov 01 21:26:57.604943 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:57.604911    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7249f4/simulator-single-main-7249f4-kserve-7cb846fc58-xqfnp"]
Nov 01 21:26:58.975870 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:58.975831    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:26:59.077792 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:26:59.077756    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe11e5b-caae-44bf-902c-24870cfb8b50" path="/var/lib/kubelet/pods/7fe11e5b-caae-44bf-902c-24870cfb8b50/volumes"
Nov 01 21:27:08.976311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:08.976274    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:27:14.349159 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.349129    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:27:14.514671 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.514620    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume\") pod \"9eccb57f-3641-4004-974d-a803bbbeb19c\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") "
Nov 01 21:27:14.514842 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.514748    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd\" (UniqueName: \"kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd\") pod \"9eccb57f-3641-4004-974d-a803bbbeb19c\" (UID: \"9eccb57f-3641-4004-974d-a803bbbeb19c\") "
Nov 01 21:27:14.515017 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.514995    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume" pod "9eccb57f-3641-4004-974d-a803bbbeb19c" (UID: "9eccb57f-3641-4004-974d-a803bbbeb19c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:27:14.516778 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.516756    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd" pod "9eccb57f-3641-4004-974d-a803bbbeb19c" (UID: "9eccb57f-3641-4004-974d-a803bbbeb19c"). InnerVolumeSpecName "kube-api-access-dgmcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:27:14.537754 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.537726    4872 generic.go:356] "Generic (PLEG): container finished" podID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerID="2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24" exitCode=0
Nov 01 21:27:14.537890 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.537776    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"
Nov 01 21:27:14.537890 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.537786    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" event={"ID":"9eccb57f-3641-4004-974d-a803bbbeb19c","Type":"ContainerDied","Data":"2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24"}
Nov 01 21:27:14.537890 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.537872    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s" event={"ID":"9eccb57f-3641-4004-974d-a803bbbeb19c","Type":"ContainerDied","Data":"c759183c8dc9aa4fe673c9e671e707e9cabdde727535fc215bb4e9ed8e3caee5"}
Nov 01 21:27:14.538107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.537903    4872 scope.go:122] "RemoveContainer" containerID="2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24"
Nov 01 21:27:14.547087 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.547070    4872 scope.go:122] "RemoveContainer" containerID="2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24"
Nov 01 21:27:14.547334 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:27:14.547318    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24\": container with ID starting with 2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24 not found: ID does not exist" containerID="2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24"
Nov 01 21:27:14.547382 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.547342    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24"} err="failed to get container status \"2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24\": rpc error: code = NotFound desc = could not find container \"2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24\": container with ID starting with 2479f7cdb8191f94e898cc49fa72c562d6eee6de20abb6094de19850f8874f24 not found: ID does not exist"
Nov 01 21:27:14.615467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.615435    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmcd\" (UniqueName: \"kubernetes.io/projected/9eccb57f-3641-4004-974d-a803bbbeb19c-kube-api-access-dgmcd\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:27:14.615467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.615465    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eccb57f-3641-4004-974d-a803bbbeb19c-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:27:14.703171 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.703139    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"]
Nov 01 21:27:14.707582 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:14.707557    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7249f4-7568bd48ff597s"]
Nov 01 21:27:15.077434 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:27:15.077403    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" path="/var/lib/kubelet/pods/9eccb57f-3641-4004-974d-a803bbbeb19c/volumes"
Nov 01 21:27:23.389267 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:27:23.389209    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:27:53.364815 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:27:53.364770    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:28:23.365432 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:28:23.365391    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:28:53.364233 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:28:53.364193    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:29:23.364414 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:29:23.364378    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:29:53.363838 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:29:53.363801    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:30:03.035062 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035027    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"]
Nov 01 21:30:03.035519 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035503    4872 cpu_manager.go:414] "containerMap: removing container" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerName="payload-processing"
Nov 01 21:30:03.035567 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035522    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerName="payload-processing"
Nov 01 21:30:03.035567 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035550    4872 cpu_manager.go:414] "containerMap: removing container" podUID="7fe11e5b-caae-44bf-902c-24870cfb8b50" containerName="main"
Nov 01 21:30:03.035567 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035556    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe11e5b-caae-44bf-902c-24870cfb8b50" containerName="main"
Nov 01 21:30:03.035567 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035564    4872 cpu_manager.go:414] "containerMap: removing container" podUID="7204442b-7399-4f79-a50e-2073fdb73704" containerName="maas-api"
Nov 01 21:30:03.035705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035571    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7204442b-7399-4f79-a50e-2073fdb73704" containerName="maas-api"
Nov 01 21:30:03.035705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035637    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe11e5b-caae-44bf-902c-24870cfb8b50" containerName="main"
Nov 01 21:30:03.035705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035647    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="7204442b-7399-4f79-a50e-2073fdb73704" containerName="maas-api"
Nov 01 21:30:03.035705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.035671    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eccb57f-3641-4004-974d-a803bbbeb19c" containerName="payload-processing"
Nov 01 21:30:03.038879 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.038863    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.096772 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.096742    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.097026 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.097000    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwb9m\" (UniqueName: \"kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m\") pod \"payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.143021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.142986    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-236f0f\""
Nov 01 21:30:03.143021 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.142986    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9ce98c44-dockercfg-v5pkt\""
Nov 01 21:30:03.144876 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.144839    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"]
Nov 01 21:30:03.198327 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.198299    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwb9m\" (UniqueName: \"kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m\") pod \"payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.198498 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.198375    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.199092 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.199071    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.255405 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.255372    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwb9m\" (UniqueName: \"kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m\") pod \"payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.352933 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.352836    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:03.725168 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:03.725142    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"]
Nov 01 21:30:03.727438 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:30:03.727398    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e8ffdb_bbbb_4401_90eb_1d0d045e56f7.slice/crio-3f37dbec45b7c632d3515bc1872ad2f581ee618c8db2bf0f88c06dc27084ccc7 WatchSource:0}: Error finding container 3f37dbec45b7c632d3515bc1872ad2f581ee618c8db2bf0f88c06dc27084ccc7: Status 404 returned error can't find the container with id 3f37dbec45b7c632d3515bc1872ad2f581ee618c8db2bf0f88c06dc27084ccc7
Nov 01 21:30:04.328837 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:04.328804    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" event={"ID":"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7","Type":"ContainerStarted","Data":"41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7"}
Nov 01 21:30:04.329263 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:04.328851    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" event={"ID":"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7","Type":"ContainerStarted","Data":"3f37dbec45b7c632d3515bc1872ad2f581ee618c8db2bf0f88c06dc27084ccc7"}
Nov 01 21:30:04.329263 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:04.328907    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:04.445326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:04.445273    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" podStartSLOduration=2.44525845 podStartE2EDuration="2.44525845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:04.442738605 +0000 UTC m=+3561.491848422" watchObservedRunningTime="2026-11-01 21:30:04.44525845 +0000 UTC m=+3561.494368316"
Nov 01 21:30:05.248788 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.248757    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"]
Nov 01 21:30:05.252315 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.252300    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.318533 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.318490    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.318740 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.318560    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.318740 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.318698    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.318871 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.318848    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rt9t\" (UniqueName: \"kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.357584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.357551    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-236f0f-serving-cert\""
Nov 01 21:30:05.359573 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.359553    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"]
Nov 01 21:30:05.419506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.419467    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.419702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.419609    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rt9t\" (UniqueName: \"kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.419702 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:30:05.419628    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-serving-cert: secret "maas-api-e2e-worker-single-main-236f0f-serving-cert" not found
Nov 01 21:30:05.419702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.419689    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.419892 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:30:05.419708    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls podName:3ecae668-fbf4-43fe-8e98-811ebe6d1961 nodeName:}" failed. No retries permitted until 2026-11-01 21:30:05.919684849 +0000 UTC m=+3562.968794660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls") pod "maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" (UID: "3ecae668-fbf4-43fe-8e98-811ebe6d1961") : secret "maas-api-e2e-worker-single-main-236f0f-serving-cert" not found
Nov 01 21:30:05.419892 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.419777    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.420421 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.420396    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.422404 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.422381    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.477922 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.477889    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rt9t\" (UniqueName: \"kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.923851 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.923819    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:05.926296 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:05.926269    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:06.165257 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:06.165226    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:06.541492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:06.541427    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"]
Nov 01 21:30:06.543848 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:30:06.543817    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecae668_fbf4_43fe_8e98_811ebe6d1961.slice/crio-35ed74a84e29d36cf01dbaffd4713b7e4debeda3b1d3920710fd2a055051c6f2 WatchSource:0}: Error finding container 35ed74a84e29d36cf01dbaffd4713b7e4debeda3b1d3920710fd2a055051c6f2: Status 404 returned error can't find the container with id 35ed74a84e29d36cf01dbaffd4713b7e4debeda3b1d3920710fd2a055051c6f2
Nov 01 21:30:07.343717 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:07.343682    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" event={"ID":"3ecae668-fbf4-43fe-8e98-811ebe6d1961","Type":"ContainerStarted","Data":"35ed74a84e29d36cf01dbaffd4713b7e4debeda3b1d3920710fd2a055051c6f2"}
Nov 01 21:30:08.349304 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:08.349267    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" event={"ID":"3ecae668-fbf4-43fe-8e98-811ebe6d1961","Type":"ContainerStarted","Data":"15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf"}
Nov 01 21:30:08.349725 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:08.349349    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:08.465993 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:08.465939    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" podStartSLOduration=2.569016341 podStartE2EDuration="3.465925148s" totalImagesPullingTime="896.908807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:08.463809847 +0000 UTC m=+3565.512919664" watchObservedRunningTime="2026-11-01 21:30:08.465925148 +0000 UTC m=+3565.515035029"
Nov 01 21:30:14.360184 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:14.360151    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:30:15.336934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:15.336907    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:30:23.364458 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:30:23.364406    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:30:27.608375 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.608339    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"]
Nov 01 21:30:27.612140 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.612119    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.717035 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.716974    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-236f0f\"/\"openshift-service-ca.crt\""
Nov 01 21:30:27.717201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.717065    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-236f0f\"/\"kube-root-ca.crt\""
Nov 01 21:30:27.717201 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.717066    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-236f0f\"/\"default-dockercfg-j9ldm\""
Nov 01 21:30:27.717338 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.717142    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-236f0f\"/\"simulator-single-main-236f0f-kserve-self-signed-certs\""
Nov 01 21:30:27.719071 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.719047    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"]
Nov 01 21:30:27.737580 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.737554    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9hr\" (UniqueName: \"kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.737729 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.737584    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.737729 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.737624    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.737819 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.737784    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.737862 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.737847    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.737920 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.737900    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.838864 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.838827    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.838889    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.838948    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.838967    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.839016    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9hr\" (UniqueName: \"kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.839036    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839365 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.839299    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839365 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.839329    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.839438 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.839389    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.841199 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.841179    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.841418 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.841399    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.896334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.896310    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9hr\" (UniqueName: \"kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr\") pod \"simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") " pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:27.923112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:27.923082    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:28.297850 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:28.297824    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"]
Nov 01 21:30:28.299447 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:30:28.299418    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b0d3bf_2901_44fa_8960_1f60bdadd365.slice/crio-305ae0b67fb5c5e9c7104041011e48012e036acbdf1b62876edd7813d5f08bfa WatchSource:0}: Error finding container 305ae0b67fb5c5e9c7104041011e48012e036acbdf1b62876edd7813d5f08bfa: Status 404 returned error can't find the container with id 305ae0b67fb5c5e9c7104041011e48012e036acbdf1b62876edd7813d5f08bfa
Nov 01 21:30:28.446227 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:28.446193    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg" event={"ID":"a9b0d3bf-2901-44fa-8960-1f60bdadd365","Type":"ContainerStarted","Data":"271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b"}
Nov 01 21:30:28.446393 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:28.446234    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg" event={"ID":"a9b0d3bf-2901-44fa-8960-1f60bdadd365","Type":"ContainerStarted","Data":"305ae0b67fb5c5e9c7104041011e48012e036acbdf1b62876edd7813d5f08bfa"}
Nov 01 21:30:28.564264 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:28.564152    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg" podStartSLOduration=1.564134756 podStartE2EDuration="1.564134756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:28.561408138 +0000 UTC m=+3585.610517956" watchObservedRunningTime="2026-11-01 21:30:28.564134756 +0000 UTC m=+3585.613244555"
Nov 01 21:30:28.923982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:28.923952    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:37.923584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:37.923547    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:37.935763 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:37.935740    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:38.508872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:38.508843    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:30:43.672340 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:43.672307    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:30:43.680354 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:43.680326    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:30:43.681381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:43.681362    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:30:43.686008 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:30:43.685990    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:30:53.366607 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:30:53.366568    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:31:23.365986 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:31:23.365948    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:31:28.181316 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:28.181234    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"]
Nov 01 21:31:35.246697 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:35.246642    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"]
Nov 01 21:31:40.247616 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:40.247570    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerName="payload-processing" containerID="cri-o://41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7" gracePeriod=25
Nov 01 21:31:41.802970 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:41.802934    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"]
Nov 01 21:31:41.803414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:41.803158    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" podUID="3ecae668-fbf4-43fe-8e98-811ebe6d1961" containerName="maas-api" containerID="cri-o://15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf" gracePeriod=30
Nov 01 21:31:43.055231 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.055206    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:31:43.202584 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.202509    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg" podUID="a9b0d3bf-2901-44fa-8960-1f60bdadd365" containerName="main" containerID="cri-o://271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b" gracePeriod=45
Nov 01 21:31:43.209870 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.209833    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t\" (UniqueName: \"kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t\") pod \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") "
Nov 01 21:31:43.210802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.209898    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\") pod \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") "
Nov 01 21:31:43.210802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.209941    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca\") pod \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") "
Nov 01 21:31:43.210802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.210006    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs\") pod \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\" (UID: \"3ecae668-fbf4-43fe-8e98-811ebe6d1961\") "
Nov 01 21:31:43.210802 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.210469    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca" pod "3ecae668-fbf4-43fe-8e98-811ebe6d1961" (UID: "3ecae668-fbf4-43fe-8e98-811ebe6d1961"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:31:43.212338 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.212307    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls" pod "3ecae668-fbf4-43fe-8e98-811ebe6d1961" (UID: "3ecae668-fbf4-43fe-8e98-811ebe6d1961"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:31:43.212472 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.212456    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t" pod "3ecae668-fbf4-43fe-8e98-811ebe6d1961" (UID: "3ecae668-fbf4-43fe-8e98-811ebe6d1961"). InnerVolumeSpecName "kube-api-access-2rt9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:31:43.212641 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.212615    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs" pod "3ecae668-fbf4-43fe-8e98-811ebe6d1961" (UID: "3ecae668-fbf4-43fe-8e98-811ebe6d1961"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:31:43.311452 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.311412    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rt9t\" (UniqueName: \"kubernetes.io/projected/3ecae668-fbf4-43fe-8e98-811ebe6d1961-kube-api-access-2rt9t\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.311452 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.311444    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.311452 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.311454    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecae668-fbf4-43fe-8e98-811ebe6d1961-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.311452 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.311464    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ecae668-fbf4-43fe-8e98-811ebe6d1961-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.448322 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.448296    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:31:43.614239 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614152    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir\") pod \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") "
Nov 01 21:31:43.614406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614276    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs\") pod \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") "
Nov 01 21:31:43.614406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614307    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr\" (UniqueName: \"kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr\") pod \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") "
Nov 01 21:31:43.614406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614346    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home\") pod \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") "
Nov 01 21:31:43.614406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614374    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache\") pod \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") "
Nov 01 21:31:43.614406 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614402    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm\") pod \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\" (UID: \"a9b0d3bf-2901-44fa-8960-1f60bdadd365\") "
Nov 01 21:31:43.614710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614529    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir" pod "a9b0d3bf-2901-44fa-8960-1f60bdadd365" (UID: "a9b0d3bf-2901-44fa-8960-1f60bdadd365"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:31:43.614710 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614623    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home" pod "a9b0d3bf-2901-44fa-8960-1f60bdadd365" (UID: "a9b0d3bf-2901-44fa-8960-1f60bdadd365"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:31:43.614794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614763    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.614794 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614788    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.614905 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.614835    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache" pod "a9b0d3bf-2901-44fa-8960-1f60bdadd365" (UID: "a9b0d3bf-2901-44fa-8960-1f60bdadd365"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:31:43.616803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.616772    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs" pod "a9b0d3bf-2901-44fa-8960-1f60bdadd365" (UID: "a9b0d3bf-2901-44fa-8960-1f60bdadd365"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:31:43.616958 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.616801    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm" pod "a9b0d3bf-2901-44fa-8960-1f60bdadd365" (UID: "a9b0d3bf-2901-44fa-8960-1f60bdadd365"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:31:43.617015 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.616958    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr" pod "a9b0d3bf-2901-44fa-8960-1f60bdadd365" (UID: "a9b0d3bf-2901-44fa-8960-1f60bdadd365"). InnerVolumeSpecName "kube-api-access-cj9hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:31:43.715702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.715620    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.715702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.715696    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b0d3bf-2901-44fa-8960-1f60bdadd365-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.715702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.715706    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b0d3bf-2901-44fa-8960-1f60bdadd365-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.715702 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.715717    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj9hr\" (UniqueName: \"kubernetes.io/projected/a9b0d3bf-2901-44fa-8960-1f60bdadd365-kube-api-access-cj9hr\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:43.805966 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.805931    4872 generic.go:356] "Generic (PLEG): container finished" podID="a9b0d3bf-2901-44fa-8960-1f60bdadd365" containerID="271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b" exitCode=0
Nov 01 21:31:43.806176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.805975    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"
Nov 01 21:31:43.806176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.806018    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg" event={"ID":"a9b0d3bf-2901-44fa-8960-1f60bdadd365","Type":"ContainerDied","Data":"271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b"}
Nov 01 21:31:43.806176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.806059    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg" event={"ID":"a9b0d3bf-2901-44fa-8960-1f60bdadd365","Type":"ContainerDied","Data":"305ae0b67fb5c5e9c7104041011e48012e036acbdf1b62876edd7813d5f08bfa"}
Nov 01 21:31:43.806176 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.806075    4872 scope.go:122] "RemoveContainer" containerID="271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b"
Nov 01 21:31:43.807589 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.807562    4872 generic.go:356] "Generic (PLEG): container finished" podID="3ecae668-fbf4-43fe-8e98-811ebe6d1961" containerID="15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf" exitCode=0
Nov 01 21:31:43.807715 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.807608    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" event={"ID":"3ecae668-fbf4-43fe-8e98-811ebe6d1961","Type":"ContainerDied","Data":"15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf"}
Nov 01 21:31:43.807715 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.807639    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h" event={"ID":"3ecae668-fbf4-43fe-8e98-811ebe6d1961","Type":"ContainerDied","Data":"35ed74a84e29d36cf01dbaffd4713b7e4debeda3b1d3920710fd2a055051c6f2"}
Nov 01 21:31:43.807816 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.807722    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"
Nov 01 21:31:43.816197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.816099    4872 scope.go:122] "RemoveContainer" containerID="271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b"
Nov 01 21:31:43.816470 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:31:43.816449    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b\": container with ID starting with 271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b not found: ID does not exist" containerID="271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b"
Nov 01 21:31:43.816546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.816477    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b"} err="failed to get container status \"271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b\": rpc error: code = NotFound desc = could not find container \"271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b\": container with ID starting with 271ae03fc8d71bfb17afedca799d8dcb37d7d591defd828c0d521c60a5056c8b not found: ID does not exist"
Nov 01 21:31:43.816546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.816496    4872 scope.go:122] "RemoveContainer" containerID="15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf"
Nov 01 21:31:43.825813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.825793    4872 scope.go:122] "RemoveContainer" containerID="15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf"
Nov 01 21:31:43.826116 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:31:43.826093    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf\": container with ID starting with 15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf not found: ID does not exist" containerID="15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf"
Nov 01 21:31:43.826169 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.826127    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf"} err="failed to get container status \"15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf\": rpc error: code = NotFound desc = could not find container \"15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf\": container with ID starting with 15b63394f381265e0471f53a3b1d34c44fa477024c4938b27959fed2e2be2ddf not found: ID does not exist"
Nov 01 21:31:43.970926 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:43.970892    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"]
Nov 01 21:31:44.021830 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:44.021790    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-236f0f/simulator-single-main-236f0f-kserve-68756d4dbc-t2mxg"]
Nov 01 21:31:44.084340 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:44.084299    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"]
Nov 01 21:31:44.088388 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:44.088352    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-236f0f-74446576c-5fz8h"]
Nov 01 21:31:45.078046 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:45.078010    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecae668-fbf4-43fe-8e98-811ebe6d1961" path="/var/lib/kubelet/pods/3ecae668-fbf4-43fe-8e98-811ebe6d1961/volumes"
Nov 01 21:31:45.078442 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:45.078428    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b0d3bf-2901-44fa-8960-1f60bdadd365" path="/var/lib/kubelet/pods/a9b0d3bf-2901-44fa-8960-1f60bdadd365/volumes"
Nov 01 21:31:45.334415 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:45.334334    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:31:53.369042 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:31:53.369002    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:31:55.335233 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:55.335190    4872 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:31:56.370776 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.370736    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"]
Nov 01 21:31:56.371253 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.371238    4872 cpu_manager.go:414] "containerMap: removing container" podUID="a9b0d3bf-2901-44fa-8960-1f60bdadd365" containerName="main"
Nov 01 21:31:56.371323 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.371254    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b0d3bf-2901-44fa-8960-1f60bdadd365" containerName="main"
Nov 01 21:31:56.371323 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.371277    4872 cpu_manager.go:414] "containerMap: removing container" podUID="3ecae668-fbf4-43fe-8e98-811ebe6d1961" containerName="maas-api"
Nov 01 21:31:56.371323 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.371283    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecae668-fbf4-43fe-8e98-811ebe6d1961" containerName="maas-api"
Nov 01 21:31:56.371422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.371356    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b0d3bf-2901-44fa-8960-1f60bdadd365" containerName="main"
Nov 01 21:31:56.371422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.371369    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecae668-fbf4-43fe-8e98-811ebe6d1961" containerName="maas-api"
Nov 01 21:31:56.374539 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.374524    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.439949 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.439912    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.440136 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.440044    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.440136 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.440091    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.440136 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.440115    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.440297 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.440271    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.440359 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.440341    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9rk\" (UniqueName: \"kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.478876 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.478839    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-835c913f-kserve-self-signed-certs\""
Nov 01 21:31:56.480914 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.480893    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"]
Nov 01 21:31:56.541323 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541283    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541528 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541400    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541528 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541430    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541528 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541454    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541528 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541507    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541560    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9rk\" (UniqueName: \"kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541895    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.541934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541928    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.542137 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.541973    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.543740 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.543718    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.544000 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.543985    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.599055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.599024    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9rk\" (UniqueName: \"kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk\") pod \"e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") " pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:56.686268 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:56.686236    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:31:57.058947 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:57.058919    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"]
Nov 01 21:31:57.061539 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:31:57.061509    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca605dec_8b62_4706_bc0c_0ae108cd27e4.slice/crio-7d715ba98aee9bb9c79ff9eeeb33958a057b8d1d4d3905c0621e58020fbcea0a WatchSource:0}: Error finding container 7d715ba98aee9bb9c79ff9eeeb33958a057b8d1d4d3905c0621e58020fbcea0a: Status 404 returned error can't find the container with id 7d715ba98aee9bb9c79ff9eeeb33958a057b8d1d4d3905c0621e58020fbcea0a
Nov 01 21:31:57.876025 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:57.875984    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7" event={"ID":"ca605dec-8b62-4706-bc0c-0ae108cd27e4","Type":"ContainerStarted","Data":"013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6"}
Nov 01 21:31:57.876429 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:57.876037    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7" event={"ID":"ca605dec-8b62-4706-bc0c-0ae108cd27e4","Type":"ContainerStarted","Data":"7d715ba98aee9bb9c79ff9eeeb33958a057b8d1d4d3905c0621e58020fbcea0a"}
Nov 01 21:31:57.991267 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:57.991207    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7" podStartSLOduration=1.99118869 podStartE2EDuration="1.99118869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:31:57.990292339 +0000 UTC m=+3675.039402180" watchObservedRunningTime="2026-11-01 21:31:57.99118869 +0000 UTC m=+3675.040298510"
Nov 01 21:31:58.687010 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:31:58.686974    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:32:00.507827 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.507760    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:32:00.581030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.580997    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume\") pod \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") "
Nov 01 21:32:00.581224 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.581066    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m\" (UniqueName: \"kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m\") pod \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\" (UID: \"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7\") "
Nov 01 21:32:00.581467 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.581436    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume" pod "27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" (UID: "27e8ffdb-bbbb-4401-90eb-1d0d045e56f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:32:00.583166 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.583142    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m" pod "27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" (UID: "27e8ffdb-bbbb-4401-90eb-1d0d045e56f7"). InnerVolumeSpecName "kube-api-access-bwb9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:32:00.682606 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.682562    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:32:00.682606 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.682596    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwb9m\" (UniqueName: \"kubernetes.io/projected/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7-kube-api-access-bwb9m\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:32:00.892074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.891985    4872 generic.go:356] "Generic (PLEG): container finished" podID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerID="41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7" exitCode=0
Nov 01 21:32:00.892074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.892033    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"
Nov 01 21:32:00.892333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.892073    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" event={"ID":"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7","Type":"ContainerDied","Data":"41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7"}
Nov 01 21:32:00.892333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.892134    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97" event={"ID":"27e8ffdb-bbbb-4401-90eb-1d0d045e56f7","Type":"ContainerDied","Data":"3f37dbec45b7c632d3515bc1872ad2f581ee618c8db2bf0f88c06dc27084ccc7"}
Nov 01 21:32:00.892333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.892163    4872 scope.go:122] "RemoveContainer" containerID="41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7"
Nov 01 21:32:00.901863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.901847    4872 scope.go:122] "RemoveContainer" containerID="41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7"
Nov 01 21:32:00.902118 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:32:00.902103    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7\": container with ID starting with 41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7 not found: ID does not exist" containerID="41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7"
Nov 01 21:32:00.902189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:00.902126    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7"} err="failed to get container status \"41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7\": rpc error: code = NotFound desc = could not find container \"41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7\": container with ID starting with 41a9a0471b9d48aba68f32909204205701e7be04148e5b7f00ce5153950459f7 not found: ID does not exist"
Nov 01 21:32:01.060687 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:01.060639    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"]
Nov 01 21:32:01.110907 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:01.110876    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-236f0f-58779bccbs4r97"]
Nov 01 21:32:03.079206 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:03.079172    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" path="/var/lib/kubelet/pods/27e8ffdb-bbbb-4401-90eb-1d0d045e56f7/volumes"
Nov 01 21:32:06.686470 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:06.686422    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:32:06.699478 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:06.699448    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:32:06.953136 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:06.953050    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:32:23.363288 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:32:23.363253    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:32:41.011397 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.011366    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"]
Nov 01 21:32:41.011840 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.011810    4872 cpu_manager.go:414] "containerMap: removing container" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerName="payload-processing"
Nov 01 21:32:41.011840 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.011823    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerName="payload-processing"
Nov 01 21:32:41.011918 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.011901    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e8ffdb-bbbb-4401-90eb-1d0d045e56f7" containerName="payload-processing"
Nov 01 21:32:41.015056 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.015039    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.102905 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.102874    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.103064 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.102916    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsw4\" (UniqueName: \"kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.103064 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.103018    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.103064 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.103050    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.103179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.103144    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.103179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.103172    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.120069 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.120044    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-835c913f-kserve-self-signed-certs\""
Nov 01 21:32:41.122011 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.121981    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"]
Nov 01 21:32:41.203938 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.203909    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsw4\" (UniqueName: \"kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.203957    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.203988    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.204016    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.204034    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204357 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.204127    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204468 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.204441    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204664 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.204500    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.204776 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.204523    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.206585 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.206561    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.206804 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.206783    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.261004 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.260979    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsw4\" (UniqueName: \"kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4\") pod \"e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") " pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.326972 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.326920    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:41.696093 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:41.696068    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"]
Nov 01 21:32:41.698174 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:32:41.698123    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4c2253e_79c1_4109_8d16_21a855e5eead.slice/crio-cf89780f6f5217d0db514f12f6f0ee8f9f7112a16da7730f7f923e613b23d010 WatchSource:0}: Error finding container cf89780f6f5217d0db514f12f6f0ee8f9f7112a16da7730f7f923e613b23d010: Status 404 returned error can't find the container with id cf89780f6f5217d0db514f12f6f0ee8f9f7112a16da7730f7f923e613b23d010
Nov 01 21:32:42.108537 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:42.108446    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq" event={"ID":"b4c2253e-79c1-4109-8d16-21a855e5eead","Type":"ContainerStarted","Data":"cdb357411599538ca465a85b7a34146bcbfbab632fa248c80011f1aff1f51e3d"}
Nov 01 21:32:42.108537 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:42.108484    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq" event={"ID":"b4c2253e-79c1-4109-8d16-21a855e5eead","Type":"ContainerStarted","Data":"cf89780f6f5217d0db514f12f6f0ee8f9f7112a16da7730f7f923e613b23d010"}
Nov 01 21:32:42.223909 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:42.223853    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq" podStartSLOduration=2.223838708 podStartE2EDuration="2.223838708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:32:42.222921603 +0000 UTC m=+3719.272031428" watchObservedRunningTime="2026-11-01 21:32:42.223838708 +0000 UTC m=+3719.272948558"
Nov 01 21:32:42.327185 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:42.327158    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:51.327561 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:51.327526    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:51.339788 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:51.339764    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:52.192519 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:32:52.192490    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:32:53.364689 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:32:53.364632    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:33:15.632833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:15.632756    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"]
Nov 01 21:33:23.364390 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:33:23.364357    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:33:30.280781 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.280752    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"]
Nov 01 21:33:30.651856 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.651816    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7" podUID="ca605dec-8b62-4706-bc0c-0ae108cd27e4" containerName="main" containerID="cri-o://013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6" gracePeriod=45
Nov 01 21:33:30.900269 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.900246    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:33:30.935538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935485    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs\") pod \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") "
Nov 01 21:33:30.935538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935527    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir\") pod \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") "
Nov 01 21:33:30.935751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935597    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache\") pod \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") "
Nov 01 21:33:30.935751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935621    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk\" (UniqueName: \"kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk\") pod \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") "
Nov 01 21:33:30.935751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935690    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home\") pod \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") "
Nov 01 21:33:30.935751 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935726    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm\") pod \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\" (UID: \"ca605dec-8b62-4706-bc0c-0ae108cd27e4\") "
Nov 01 21:33:30.935984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935832    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir" pod "ca605dec-8b62-4706-bc0c-0ae108cd27e4" (UID: "ca605dec-8b62-4706-bc0c-0ae108cd27e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:30.935984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935855    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache" pod "ca605dec-8b62-4706-bc0c-0ae108cd27e4" (UID: "ca605dec-8b62-4706-bc0c-0ae108cd27e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:30.936082 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.935976    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home" pod "ca605dec-8b62-4706-bc0c-0ae108cd27e4" (UID: "ca605dec-8b62-4706-bc0c-0ae108cd27e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:30.936147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.936118    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:30.936147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.936133    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:30.936147 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.936147    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:30.937792 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.937768    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs" pod "ca605dec-8b62-4706-bc0c-0ae108cd27e4" (UID: "ca605dec-8b62-4706-bc0c-0ae108cd27e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:33:30.937882 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.937846    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm" pod "ca605dec-8b62-4706-bc0c-0ae108cd27e4" (UID: "ca605dec-8b62-4706-bc0c-0ae108cd27e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:30.937882 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:30.937867    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk" pod "ca605dec-8b62-4706-bc0c-0ae108cd27e4" (UID: "ca605dec-8b62-4706-bc0c-0ae108cd27e4"). InnerVolumeSpecName "kube-api-access-sw9rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:33:31.037496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.037469    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw9rk\" (UniqueName: \"kubernetes.io/projected/ca605dec-8b62-4706-bc0c-0ae108cd27e4-kube-api-access-sw9rk\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:31.037496 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.037496    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca605dec-8b62-4706-bc0c-0ae108cd27e4-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:31.037647 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.037509    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca605dec-8b62-4706-bc0c-0ae108cd27e4-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:31.405334 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.405303    4872 generic.go:356] "Generic (PLEG): container finished" podID="ca605dec-8b62-4706-bc0c-0ae108cd27e4" containerID="013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6" exitCode=0
Nov 01 21:33:31.405774 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.405346    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"
Nov 01 21:33:31.405774 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.405351    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7" event={"ID":"ca605dec-8b62-4706-bc0c-0ae108cd27e4","Type":"ContainerDied","Data":"013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6"}
Nov 01 21:33:31.405774 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.405380    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7" event={"ID":"ca605dec-8b62-4706-bc0c-0ae108cd27e4","Type":"ContainerDied","Data":"7d715ba98aee9bb9c79ff9eeeb33958a057b8d1d4d3905c0621e58020fbcea0a"}
Nov 01 21:33:31.405774 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.405395    4872 scope.go:122] "RemoveContainer" containerID="013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6"
Nov 01 21:33:31.413883 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.413859    4872 scope.go:122] "RemoveContainer" containerID="013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6"
Nov 01 21:33:31.414163 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:33:31.414143    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6\": container with ID starting with 013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6 not found: ID does not exist" containerID="013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6"
Nov 01 21:33:31.414222 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.414170    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6"} err="failed to get container status \"013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6\": rpc error: code = NotFound desc = could not find container \"013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6\": container with ID starting with 013f37d928cc11f137a9a6de863cb2c4aeb356218049569afd4f1783764fbbd6 not found: ID does not exist"
Nov 01 21:33:31.569812 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.569789    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"]
Nov 01 21:33:31.572940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:31.572915    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-835c913f-kserve-758fbc8657-rdcq7"]
Nov 01 21:33:33.078448 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:33.078419    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca605dec-8b62-4706-bc0c-0ae108cd27e4" path="/var/lib/kubelet/pods/ca605dec-8b62-4706-bc0c-0ae108cd27e4/volumes"
Nov 01 21:33:45.299354 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.299317    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq" podUID="b4c2253e-79c1-4109-8d16-21a855e5eead" containerName="main" containerID="cri-o://cdb357411599538ca465a85b7a34146bcbfbab632fa248c80011f1aff1f51e3d" gracePeriod=45
Nov 01 21:33:45.494971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.494935    4872 generic.go:356] "Generic (PLEG): container finished" podID="b4c2253e-79c1-4109-8d16-21a855e5eead" containerID="cdb357411599538ca465a85b7a34146bcbfbab632fa248c80011f1aff1f51e3d" exitCode=0
Nov 01 21:33:45.495100 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.494984    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq" event={"ID":"b4c2253e-79c1-4109-8d16-21a855e5eead","Type":"ContainerDied","Data":"cdb357411599538ca465a85b7a34146bcbfbab632fa248c80011f1aff1f51e3d"}
Nov 01 21:33:45.550255 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.550200    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:33:45.657860 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.657835    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4\" (UniqueName: \"kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4\") pod \"b4c2253e-79c1-4109-8d16-21a855e5eead\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") "
Nov 01 21:33:45.658027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.657867    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs\") pod \"b4c2253e-79c1-4109-8d16-21a855e5eead\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") "
Nov 01 21:33:45.658027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.657929    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache\") pod \"b4c2253e-79c1-4109-8d16-21a855e5eead\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") "
Nov 01 21:33:45.658027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.657951    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home\") pod \"b4c2253e-79c1-4109-8d16-21a855e5eead\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") "
Nov 01 21:33:45.658027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.657971    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm\") pod \"b4c2253e-79c1-4109-8d16-21a855e5eead\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") "
Nov 01 21:33:45.658260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658079    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir\") pod \"b4c2253e-79c1-4109-8d16-21a855e5eead\" (UID: \"b4c2253e-79c1-4109-8d16-21a855e5eead\") "
Nov 01 21:33:45.658260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658228    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home" pod "b4c2253e-79c1-4109-8d16-21a855e5eead" (UID: "b4c2253e-79c1-4109-8d16-21a855e5eead"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:45.658260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658242    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache" pod "b4c2253e-79c1-4109-8d16-21a855e5eead" (UID: "b4c2253e-79c1-4109-8d16-21a855e5eead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:45.658414 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658373    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir" pod "b4c2253e-79c1-4109-8d16-21a855e5eead" (UID: "b4c2253e-79c1-4109-8d16-21a855e5eead"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:45.658490 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658474    4872 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-tmp-dir\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:45.658546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658491    4872 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-model-cache\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:45.658546 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.658502    4872 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-home\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:45.659982 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.659956    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm" pod "b4c2253e-79c1-4109-8d16-21a855e5eead" (UID: "b4c2253e-79c1-4109-8d16-21a855e5eead"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:33:45.660106 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.660038    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs" pod "b4c2253e-79c1-4109-8d16-21a855e5eead" (UID: "b4c2253e-79c1-4109-8d16-21a855e5eead"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:33:45.660169 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.660115    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4" pod "b4c2253e-79c1-4109-8d16-21a855e5eead" (UID: "b4c2253e-79c1-4109-8d16-21a855e5eead"). InnerVolumeSpecName "kube-api-access-6bsw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:33:45.759898 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.759873    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bsw4\" (UniqueName: \"kubernetes.io/projected/b4c2253e-79c1-4109-8d16-21a855e5eead-kube-api-access-6bsw4\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:45.759898 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.759896    4872 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c2253e-79c1-4109-8d16-21a855e5eead-tls-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:45.760045 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:45.759907    4872 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c2253e-79c1-4109-8d16-21a855e5eead-dshm\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:46.500818 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:46.500792    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"
Nov 01 21:33:46.501260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:46.500791    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq" event={"ID":"b4c2253e-79c1-4109-8d16-21a855e5eead","Type":"ContainerDied","Data":"cf89780f6f5217d0db514f12f6f0ee8f9f7112a16da7730f7f923e613b23d010"}
Nov 01 21:33:46.501260 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:46.500918    4872 scope.go:122] "RemoveContainer" containerID="cdb357411599538ca465a85b7a34146bcbfbab632fa248c80011f1aff1f51e3d"
Nov 01 21:33:46.666699 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:46.666672    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"]
Nov 01 21:33:46.717335 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:46.717309    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-835c913f-kserve-74b94dbb9c-wz5pq"]
Nov 01 21:33:47.083276 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:33:47.083248    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c2253e-79c1-4109-8d16-21a855e5eead" path="/var/lib/kubelet/pods/b4c2253e-79c1-4109-8d16-21a855e5eead/volumes"
Nov 01 21:33:53.469985 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:33:53.469866    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:34:20.878764 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.878731    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fb7d7759-5962m"]
Nov 01 21:34:20.879333 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.879313    4872 cpu_manager.go:414] "containerMap: removing container" podUID="ca605dec-8b62-4706-bc0c-0ae108cd27e4" containerName="main"
Nov 01 21:34:20.879408 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.879335    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca605dec-8b62-4706-bc0c-0ae108cd27e4" containerName="main"
Nov 01 21:34:20.879408 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.879362    4872 cpu_manager.go:414] "containerMap: removing container" podUID="b4c2253e-79c1-4109-8d16-21a855e5eead" containerName="main"
Nov 01 21:34:20.879408 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.879367    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c2253e-79c1-4109-8d16-21a855e5eead" containerName="main"
Nov 01 21:34:20.879517 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.879461    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c2253e-79c1-4109-8d16-21a855e5eead" containerName="main"
Nov 01 21:34:20.879517 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.879479    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca605dec-8b62-4706-bc0c-0ae108cd27e4" containerName="main"
Nov 01 21:34:20.882695 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.882678    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:20.988014 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:20.987981    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fb7d7759-5962m"]
Nov 01 21:34:21.025139 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.025108    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02f9e85d-56dc-4207-be48-b8b68211cf17-maas-api-tls\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.025362 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.025164    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lnpl\" (UniqueName: \"kubernetes.io/projected/02f9e85d-56dc-4207-be48-b8b68211cf17-kube-api-access-4lnpl\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.025362 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.025261    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9e85d-56dc-4207-be48-b8b68211cf17-metrics-certs\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.025362 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.025310    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f9e85d-56dc-4207-be48-b8b68211cf17-openshift-service-ca\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.125874 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.125839    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lnpl\" (UniqueName: \"kubernetes.io/projected/02f9e85d-56dc-4207-be48-b8b68211cf17-kube-api-access-4lnpl\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.126074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.125890    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9e85d-56dc-4207-be48-b8b68211cf17-metrics-certs\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.126074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.125931    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f9e85d-56dc-4207-be48-b8b68211cf17-openshift-service-ca\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.126074 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.126051    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02f9e85d-56dc-4207-be48-b8b68211cf17-maas-api-tls\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.126945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.126912    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f9e85d-56dc-4207-be48-b8b68211cf17-openshift-service-ca\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.128837 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.128768    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9e85d-56dc-4207-be48-b8b68211cf17-metrics-certs\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.128965 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.128950    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02f9e85d-56dc-4207-be48-b8b68211cf17-maas-api-tls\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.184144 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.184115    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lnpl\" (UniqueName: \"kubernetes.io/projected/02f9e85d-56dc-4207-be48-b8b68211cf17-kube-api-access-4lnpl\") pod \"maas-api-fb7d7759-5962m\" (UID: \"02f9e85d-56dc-4207-be48-b8b68211cf17\") " pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.197327 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.197301    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:21.569836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.569800    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fb7d7759-5962m"]
Nov 01 21:34:21.573989 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:34:21.573947    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f9e85d_56dc_4207_be48_b8b68211cf17.slice/crio-c7d998df24d19fe7e505b66ddc127ed3f686e7e9e72a0f892f31047f35fb2741 WatchSource:0}: Error finding container c7d998df24d19fe7e505b66ddc127ed3f686e7e9e72a0f892f31047f35fb2741: Status 404 returned error can't find the container with id c7d998df24d19fe7e505b66ddc127ed3f686e7e9e72a0f892f31047f35fb2741
Nov 01 21:34:21.575257 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.575240    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:34:21.691351 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:21.691319    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m" event={"ID":"02f9e85d-56dc-4207-be48-b8b68211cf17","Type":"ContainerStarted","Data":"c7d998df24d19fe7e505b66ddc127ed3f686e7e9e72a0f892f31047f35fb2741"}
Nov 01 21:34:22.698124 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:22.698087    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m" event={"ID":"02f9e85d-56dc-4207-be48-b8b68211cf17","Type":"ContainerStarted","Data":"8111b2b22d06266e42e29783c43fc1ebf017ca42a55a7776892c05ecd0258eaa"}
Nov 01 21:34:22.698532 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:22.698155    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:22.814626 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:22.814530    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m" podStartSLOduration=1.9303384019999998 podStartE2EDuration="2.81451641s" totalImagesPullingTime="884.178008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:22.814276817 +0000 UTC m=+3819.863386636" watchObservedRunningTime="2026-11-01 21:34:22.81451641 +0000 UTC m=+3819.863626227"
Nov 01 21:34:23.364549 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:34:23.364446    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:34:28.708731 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:28.708648    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fb7d7759-5962m"
Nov 01 21:34:35.770261 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.770231    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"]
Nov 01 21:34:35.773923 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.773904    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.826643 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.826618    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49c82ebd-serving-cert\""
Nov 01 21:34:35.859027 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.859003    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.859155 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.859048    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgz7\" (UniqueName: \"kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.859197 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.859169    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.859242 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.859231    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.869395 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.869370    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"]
Nov 01 21:34:35.960002 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.959970    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.960189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.960023    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.960189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.960066    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.960189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.960098    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgz7\" (UniqueName: \"kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.960189 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:34:35.960176    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-serving-cert: secret "maas-api-e2e-ait-49c82ebd-serving-cert" not found
Nov 01 21:34:35.960353 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:34:35.960247    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls podName:28593138-a5a0-4f30-bdfe-9870848ccaac nodeName:}" failed. No retries permitted until 2026-11-01 21:34:36.460232592 +0000 UTC m=+3833.509342401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls") pod "maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" (UID: "28593138-a5a0-4f30-bdfe-9870848ccaac") : secret "maas-api-e2e-ait-49c82ebd-serving-cert" not found
Nov 01 21:34:35.960679 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.960639    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:35.962461 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:35.962441    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:36.018366 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:36.018331    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgz7\" (UniqueName: \"kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:36.464485 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:36.464453    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:36.466863 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:36.466838    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\") pod \"maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:36.686469 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:36.686439    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:37.012037 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:37.011952    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"]
Nov 01 21:34:37.012422 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:37.012209    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2" podUID="963e514d-5cb6-4a7c-b70f-2c8592186f04" containerName="maas-api" containerID="cri-o://8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17" gracePeriod=30
Nov 01 21:34:37.058021 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:34:37.057988    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28593138_a5a0_4f30_bdfe_9870848ccaac.slice/crio-b525629cfa36a1fccbeee764341a954115c9ff8180661366c87eabb37501f909 WatchSource:0}: Error finding container b525629cfa36a1fccbeee764341a954115c9ff8180661366c87eabb37501f909: Status 404 returned error can't find the container with id b525629cfa36a1fccbeee764341a954115c9ff8180661366c87eabb37501f909
Nov 01 21:34:37.061536 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:37.061504    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"]
Nov 01 21:34:37.771940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:37.771901    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" event={"ID":"28593138-a5a0-4f30-bdfe-9870848ccaac","Type":"ContainerStarted","Data":"b525629cfa36a1fccbeee764341a954115c9ff8180661366c87eabb37501f909"}
Nov 01 21:34:38.258013 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.257988    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 21:34:38.283518 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.283491    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls\") pod \"963e514d-5cb6-4a7c-b70f-2c8592186f04\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") "
Nov 01 21:34:38.283628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.283545    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs\") pod \"963e514d-5cb6-4a7c-b70f-2c8592186f04\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") "
Nov 01 21:34:38.283628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.283580    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca\") pod \"963e514d-5cb6-4a7c-b70f-2c8592186f04\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") "
Nov 01 21:34:38.283628 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.283601    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw\" (UniqueName: \"kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw\") pod \"963e514d-5cb6-4a7c-b70f-2c8592186f04\" (UID: \"963e514d-5cb6-4a7c-b70f-2c8592186f04\") "
Nov 01 21:34:38.283980 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.283952    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca" pod "963e514d-5cb6-4a7c-b70f-2c8592186f04" (UID: "963e514d-5cb6-4a7c-b70f-2c8592186f04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:34:38.285799 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.285772    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs" pod "963e514d-5cb6-4a7c-b70f-2c8592186f04" (UID: "963e514d-5cb6-4a7c-b70f-2c8592186f04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:34:38.285900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.285775    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw" pod "963e514d-5cb6-4a7c-b70f-2c8592186f04" (UID: "963e514d-5cb6-4a7c-b70f-2c8592186f04"). InnerVolumeSpecName "kube-api-access-bb2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:34:38.285900 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.285808    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls" pod "963e514d-5cb6-4a7c-b70f-2c8592186f04" (UID: "963e514d-5cb6-4a7c-b70f-2c8592186f04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:34:38.385042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.384907    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963e514d-5cb6-4a7c-b70f-2c8592186f04-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:34:38.385042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.384939    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2tw\" (UniqueName: \"kubernetes.io/projected/963e514d-5cb6-4a7c-b70f-2c8592186f04-kube-api-access-bb2tw\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:34:38.385042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.384955    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:34:38.385042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.384968    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963e514d-5cb6-4a7c-b70f-2c8592186f04-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:34:38.779131 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.779096    4872 generic.go:356] "Generic (PLEG): container finished" podID="963e514d-5cb6-4a7c-b70f-2c8592186f04" containerID="8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17" exitCode=0
Nov 01 21:34:38.779326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.779137    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"
Nov 01 21:34:38.779326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.779178    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2" event={"ID":"963e514d-5cb6-4a7c-b70f-2c8592186f04","Type":"ContainerDied","Data":"8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17"}
Nov 01 21:34:38.779326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.779236    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2" event={"ID":"963e514d-5cb6-4a7c-b70f-2c8592186f04","Type":"ContainerDied","Data":"a9816e4c3d146c2634c0269c0423c9b34160d32a5e03d79d95282fd0d82e16ea"}
Nov 01 21:34:38.779326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.779265    4872 scope.go:122] "RemoveContainer" containerID="8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17"
Nov 01 21:34:38.780755 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.780723    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" event={"ID":"28593138-a5a0-4f30-bdfe-9870848ccaac","Type":"ContainerStarted","Data":"93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d"}
Nov 01 21:34:38.780888 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.780865    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:38.789118 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.789101    4872 scope.go:122] "RemoveContainer" containerID="8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17"
Nov 01 21:34:38.789385 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:34:38.789367    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17\": container with ID starting with 8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17 not found: ID does not exist" containerID="8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17"
Nov 01 21:34:38.789449 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.789389    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17"} err="failed to get container status \"8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17\": rpc error: code = NotFound desc = could not find container \"8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17\": container with ID starting with 8e46d5d2305391cec15840ec1d333e30cd4f86b7e94bdb96a8c18b6efd5d7c17 not found: ID does not exist"
Nov 01 21:34:38.948468 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:38.948419    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" podStartSLOduration=3.150124389 podStartE2EDuration="3.948406518s" totalImagesPullingTime="798.282129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:38.946451487 +0000 UTC m=+3835.995561306" watchObservedRunningTime="2026-11-01 21:34:38.948406518 +0000 UTC m=+3835.997516336"
Nov 01 21:34:39.058624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:39.058546    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"]
Nov 01 21:34:39.066755 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:39.066722    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f57b8b55f-bxpz2"]
Nov 01 21:34:39.078840 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:39.078811    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963e514d-5cb6-4a7c-b70f-2c8592186f04" path="/var/lib/kubelet/pods/963e514d-5cb6-4a7c-b70f-2c8592186f04/volumes"
Nov 01 21:34:44.792314 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:34:44.792284    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:34:53.364572 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:34:53.364536    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:35:23.364385 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:35:23.364348    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:35:24.394377 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.394339    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"]
Nov 01 21:35:24.394817 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.394625    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" podUID="28593138-a5a0-4f30-bdfe-9870848ccaac" containerName="maas-api" containerID="cri-o://93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d" gracePeriod=30
Nov 01 21:35:24.659913 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.659886    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:35:24.711480 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.711447    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca\") pod \"28593138-a5a0-4f30-bdfe-9870848ccaac\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") "
Nov 01 21:35:24.711689 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.711551    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\") pod \"28593138-a5a0-4f30-bdfe-9870848ccaac\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") "
Nov 01 21:35:24.711689 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.711579    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs\") pod \"28593138-a5a0-4f30-bdfe-9870848ccaac\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") "
Nov 01 21:35:24.711689 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.711616    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7\" (UniqueName: \"kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7\") pod \"28593138-a5a0-4f30-bdfe-9870848ccaac\" (UID: \"28593138-a5a0-4f30-bdfe-9870848ccaac\") "
Nov 01 21:35:24.711929 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.711900    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca" pod "28593138-a5a0-4f30-bdfe-9870848ccaac" (UID: "28593138-a5a0-4f30-bdfe-9870848ccaac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:35:24.712065 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.712045    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28593138-a5a0-4f30-bdfe-9870848ccaac-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:35:24.713816 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.713790    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7" pod "28593138-a5a0-4f30-bdfe-9870848ccaac" (UID: "28593138-a5a0-4f30-bdfe-9870848ccaac"). InnerVolumeSpecName "kube-api-access-2qgz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:35:24.713959 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.713941    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls" pod "28593138-a5a0-4f30-bdfe-9870848ccaac" (UID: "28593138-a5a0-4f30-bdfe-9870848ccaac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:35:24.714107 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.714088    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs" pod "28593138-a5a0-4f30-bdfe-9870848ccaac" (UID: "28593138-a5a0-4f30-bdfe-9870848ccaac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:35:24.813282 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.813222    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:35:24.813282 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.813281    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28593138-a5a0-4f30-bdfe-9870848ccaac-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:35:24.813484 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.813300    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qgz7\" (UniqueName: \"kubernetes.io/projected/28593138-a5a0-4f30-bdfe-9870848ccaac-kube-api-access-2qgz7\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:35:24.990785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.990699    4872 generic.go:356] "Generic (PLEG): container finished" podID="28593138-a5a0-4f30-bdfe-9870848ccaac" containerID="93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d" exitCode=0
Nov 01 21:35:24.990785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.990757    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"
Nov 01 21:35:24.990785 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.990773    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" event={"ID":"28593138-a5a0-4f30-bdfe-9870848ccaac","Type":"ContainerDied","Data":"93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d"}
Nov 01 21:35:24.991005 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.990828    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4" event={"ID":"28593138-a5a0-4f30-bdfe-9870848ccaac","Type":"ContainerDied","Data":"b525629cfa36a1fccbeee764341a954115c9ff8180661366c87eabb37501f909"}
Nov 01 21:35:24.991005 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.990851    4872 scope.go:122] "RemoveContainer" containerID="93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d"
Nov 01 21:35:24.999839 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:24.999817    4872 scope.go:122] "RemoveContainer" containerID="93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d"
Nov 01 21:35:25.000098 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:35:25.000078    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d\": container with ID starting with 93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d not found: ID does not exist" containerID="93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d"
Nov 01 21:35:25.000159 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:25.000113    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d"} err="failed to get container status \"93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d\": rpc error: code = NotFound desc = could not find container \"93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d\": container with ID starting with 93b2436830ca40a4fa5c8a2967a1af46480babc52aceebc5a996c0e8ca1d102d not found: ID does not exist"
Nov 01 21:35:25.156872 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:25.156837    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"]
Nov 01 21:35:25.159946 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:25.159922    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49c82ebd-64688bcdf8-7cmx4"]
Nov 01 21:35:27.079788 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:27.079752    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28593138-a5a0-4f30-bdfe-9870848ccaac" path="/var/lib/kubelet/pods/28593138-a5a0-4f30-bdfe-9870848ccaac/volumes"
Nov 01 21:35:37.965081 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965048    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"]
Nov 01 21:35:37.965538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965499    4872 cpu_manager.go:414] "containerMap: removing container" podUID="963e514d-5cb6-4a7c-b70f-2c8592186f04" containerName="maas-api"
Nov 01 21:35:37.965538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965511    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963e514d-5cb6-4a7c-b70f-2c8592186f04" containerName="maas-api"
Nov 01 21:35:37.965538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965530    4872 cpu_manager.go:414] "containerMap: removing container" podUID="28593138-a5a0-4f30-bdfe-9870848ccaac" containerName="maas-api"
Nov 01 21:35:37.965538 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965535    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28593138-a5a0-4f30-bdfe-9870848ccaac" containerName="maas-api"
Nov 01 21:35:37.965703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965608    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="963e514d-5cb6-4a7c-b70f-2c8592186f04" containerName="maas-api"
Nov 01 21:35:37.965703 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.965618    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="28593138-a5a0-4f30-bdfe-9870848ccaac" containerName="maas-api"
Nov 01 21:35:37.970022 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:37.970004    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.024956 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.024926    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"]
Nov 01 21:35:38.038150 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.038122    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.038288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.038221    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crjv\" (UniqueName: \"kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv\") pod \"payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.072983 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.072948    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-e23c3b\""
Nov 01 21:35:38.073143 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.072994    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-e23c3b-dockercfg-mwbft\""
Nov 01 21:35:38.139326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.139285    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.139515 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.139332    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crjv\" (UniqueName: \"kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv\") pod \"payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.140088 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.140065    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.196553 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.196525    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crjv\" (UniqueName: \"kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv\") pod \"payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.282951 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.282867    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:38.652156 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:38.652127    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"]
Nov 01 21:35:38.654296 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:35:38.654270    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7113848_b248_4ab7_996b_504e674a3e6f.slice/crio-fc57977d83b36b8588fba923fabbb56cd0e89fba01a2c508fa08d0777c415b7e WatchSource:0}: Error finding container fc57977d83b36b8588fba923fabbb56cd0e89fba01a2c508fa08d0777c415b7e: Status 404 returned error can't find the container with id fc57977d83b36b8588fba923fabbb56cd0e89fba01a2c508fa08d0777c415b7e
Nov 01 21:35:39.065803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:39.065700    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz" event={"ID":"d7113848-b248-4ab7-996b-504e674a3e6f","Type":"ContainerStarted","Data":"09b48e7b3bb90d38baf1e29abcff6d9cc150fc9696205a7d3078796c36968dea"}
Nov 01 21:35:39.066227 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:39.065823    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz" event={"ID":"d7113848-b248-4ab7-996b-504e674a3e6f","Type":"ContainerStarted","Data":"fc57977d83b36b8588fba923fabbb56cd0e89fba01a2c508fa08d0777c415b7e"}
Nov 01 21:35:39.066227 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:39.065853    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:39.181069 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:39.181015    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz" podStartSLOduration=2.18100239 podStartE2EDuration="2.18100239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:35:39.17976878 +0000 UTC m=+3896.228878610" watchObservedRunningTime="2026-11-01 21:35:39.18100239 +0000 UTC m=+3896.230112207"
Nov 01 21:35:40.963602 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:40.963565    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"]
Nov 01 21:35:40.967222 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:40.967201    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.070610 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.070565    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.070819 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.070793    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.070953 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.070930    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5sk\" (UniqueName: \"kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.071033 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.070968    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.072165 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.072140    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-e23c3b-serving-cert\""
Nov 01 21:35:41.077373 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.077348    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"]
Nov 01 21:35:41.172079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.172043    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.172275 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.172107    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5sk\" (UniqueName: \"kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.172275 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.172130    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.172275 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.172193    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.172423 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:35:41.172274    4872 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-serving-cert: secret "maas-api-e2e-authcfg-http-e23c3b-serving-cert" not found
Nov 01 21:35:41.172423 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:35:41.172346    4872 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls podName:bf399806-7e4a-47eb-adf7-9fc43502e47b nodeName:}" failed. No retries permitted until 2026-11-01 21:35:41.672325398 +0000 UTC m=+3898.721435208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls") pod "maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" (UID: "bf399806-7e4a-47eb-adf7-9fc43502e47b") : secret "maas-api-e2e-authcfg-http-e23c3b-serving-cert" not found
Nov 01 21:35:41.172877 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.172851    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.174748 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.174726    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.230984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.230915    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5sk\" (UniqueName: \"kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.676110 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.676075    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.678501 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.678481    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:41.879605 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:41.879572    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:42.255349 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:42.255320    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"]
Nov 01 21:35:42.257301 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:35:42.257269    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf399806_7e4a_47eb_adf7_9fc43502e47b.slice/crio-2e27f52a0bcc1521cb560f5b88a86f01869289fff577c17053cfcd1751129157 WatchSource:0}: Error finding container 2e27f52a0bcc1521cb560f5b88a86f01869289fff577c17053cfcd1751129157: Status 404 returned error can't find the container with id 2e27f52a0bcc1521cb560f5b88a86f01869289fff577c17053cfcd1751129157
Nov 01 21:35:43.086047 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:43.086016    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" event={"ID":"bf399806-7e4a-47eb-adf7-9fc43502e47b","Type":"ContainerStarted","Data":"2e27f52a0bcc1521cb560f5b88a86f01869289fff577c17053cfcd1751129157"}
Nov 01 21:35:43.704466 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:43.704437    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:35:43.710738 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:43.710711    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:35:43.711126 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:43.711110    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:35:43.716120 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:43.716102    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:35:44.092235 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:44.092135    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" event={"ID":"bf399806-7e4a-47eb-adf7-9fc43502e47b","Type":"ContainerStarted","Data":"2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d"}
Nov 01 21:35:44.092383 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:44.092238    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:44.235676 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:44.235609    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" podStartSLOduration=3.343664364 podStartE2EDuration="4.235595835s" totalImagesPullingTime="891.931471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:35:44.234604346 +0000 UTC m=+3901.283714165" watchObservedRunningTime="2026-11-01 21:35:44.235595835 +0000 UTC m=+3901.284705652"
Nov 01 21:35:50.073990 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:50.073953    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:35:50.101380 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:35:50.101345    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:35:53.364528 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:35:53.364495    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:36:23.364875 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:36:23.364838    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:36:53.364863 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:36:53.364826    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:37:23.363123 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:37:23.363088    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:37:53.364321 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:37:53.364288    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:38:23.364916 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:38:23.364878    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:38:53.364181 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:38:53.364145    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:39:23.364236 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:39:23.364202    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:39:27.997157 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:27.997126    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"]
Nov 01 21:39:27.999759 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:27.997342    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy" containerID="cri-o://0d255a9ba24bc2c5ee63727c465cca95710ef1138d75be06b4f3fbd37c34adf9" gracePeriod=30
Nov 01 21:39:33.144705 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.144651    4872 generic.go:356] "Generic (PLEG): container finished" podID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerID="0d255a9ba24bc2c5ee63727c465cca95710ef1138d75be06b4f3fbd37c34adf9" exitCode=0
Nov 01 21:39:33.145092 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.144724    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" event={"ID":"2637e83c-ebe9-41ba-89b1-10d57a44ff6d","Type":"ContainerDied","Data":"0d255a9ba24bc2c5ee63727c465cca95710ef1138d75be06b4f3fbd37c34adf9"}
Nov 01 21:39:33.263834 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.263809    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:39:33.318635 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318574    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.318635 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318632    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.318836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318676    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.318836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318693    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.318836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318737    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.318836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318791    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.318836 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318808    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.319091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318858    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.319091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318896    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket\") pod \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\" (UID: \"2637e83c-ebe9-41ba-89b1-10d57a44ff6d\") "
Nov 01 21:39:33.319091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.318896    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:39:33.319091 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319014    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:39:33.319293 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319201    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:39:33.319293 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319220    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:39:33.319293 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319248    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:39:33.319457 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319309    4872 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-credential-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.319457 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319330    4872 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.319457 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319345    4872 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-data\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.319457 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.319358    4872 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-workload-socket\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.320875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.320850    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:39:33.320875 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.320861    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:39:33.321180 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.321165    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "kube-api-access-fn5ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:39:33.321258 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.321237    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token" pod "2637e83c-ebe9-41ba-89b1-10d57a44ff6d" (UID: "2637e83c-ebe9-41ba-89b1-10d57a44ff6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:39:33.419842 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.419812    4872 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-podinfo\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.419842 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.419840    4872 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-envoy\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.420022 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.419849    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5ld\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-kube-api-access-fn5ld\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.420022 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.419859    4872 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istiod-ca-cert\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:33.420022 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:33.419868    4872 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637e83c-ebe9-41ba-89b1-10d57a44ff6d-istio-token\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:39:34.150643 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:34.150606    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t" event={"ID":"2637e83c-ebe9-41ba-89b1-10d57a44ff6d","Type":"ContainerDied","Data":"0591c3e292afea6b6fbc018193788a1d21717419f7627e692b899d275371181c"}
Nov 01 21:39:34.151055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:34.150678    4872 scope.go:122] "RemoveContainer" containerID="0d255a9ba24bc2c5ee63727c465cca95710ef1138d75be06b4f3fbd37c34adf9"
Nov 01 21:39:34.151055 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:34.150614    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"
Nov 01 21:39:34.368250 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:34.368216    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"]
Nov 01 21:39:34.418971 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:34.418901    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qps6t"]
Nov 01 21:39:35.077993 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:39:35.077962    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" path="/var/lib/kubelet/pods/2637e83c-ebe9-41ba-89b1-10d57a44ff6d/volumes"
Nov 01 21:39:53.363817 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:39:53.363779    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:40:16.114256 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:16.114219    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"]
Nov 01 21:40:21.114748 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.114704    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz" podUID="d7113848-b248-4ab7-996b-504e674a3e6f" containerName="payload-processing" containerID="cri-o://09b48e7b3bb90d38baf1e29abcff6d9cc150fc9696205a7d3078796c36968dea" gracePeriod=25
Nov 01 21:40:21.362708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.362677    4872 generic.go:356] "Generic (PLEG): container finished" podID="d7113848-b248-4ab7-996b-504e674a3e6f" containerID="09b48e7b3bb90d38baf1e29abcff6d9cc150fc9696205a7d3078796c36968dea" exitCode=0
Nov 01 21:40:21.362829 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.362684    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz" event={"ID":"d7113848-b248-4ab7-996b-504e674a3e6f","Type":"ContainerDied","Data":"09b48e7b3bb90d38baf1e29abcff6d9cc150fc9696205a7d3078796c36968dea"}
Nov 01 21:40:21.362869 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.362822    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz" event={"ID":"d7113848-b248-4ab7-996b-504e674a3e6f","Type":"ContainerDied","Data":"fc57977d83b36b8588fba923fabbb56cd0e89fba01a2c508fa08d0777c415b7e"}
Nov 01 21:40:21.362869 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.362846    4872 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc57977d83b36b8588fba923fabbb56cd0e89fba01a2c508fa08d0777c415b7e"
Nov 01 21:40:21.364959 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.364907    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:40:21.449945 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.449920    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume\") pod \"d7113848-b248-4ab7-996b-504e674a3e6f\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") "
Nov 01 21:40:21.450175 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.450085    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv\" (UniqueName: \"kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv\") pod \"d7113848-b248-4ab7-996b-504e674a3e6f\" (UID: \"d7113848-b248-4ab7-996b-504e674a3e6f\") "
Nov 01 21:40:21.450284 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.450262    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume" pod "d7113848-b248-4ab7-996b-504e674a3e6f" (UID: "d7113848-b248-4ab7-996b-504e674a3e6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:40:21.450378 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.450361    4872 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7113848-b248-4ab7-996b-504e674a3e6f-plugins-config-volume\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:21.451992 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.451973    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv" pod "d7113848-b248-4ab7-996b-504e674a3e6f" (UID: "d7113848-b248-4ab7-996b-504e674a3e6f"). InnerVolumeSpecName "kube-api-access-8crjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:40:21.551140 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:21.551115    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8crjv\" (UniqueName: \"kubernetes.io/projected/d7113848-b248-4ab7-996b-504e674a3e6f-kube-api-access-8crjv\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:22.367557 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:22.367519    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"
Nov 01 21:40:22.531915 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:22.531883    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"]
Nov 01 21:40:22.582151 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:22.582124    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-e23c3b-c8996fd87-qfthz"]
Nov 01 21:40:23.077881 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:23.077851    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7113848-b248-4ab7-996b-504e674a3e6f" path="/var/lib/kubelet/pods/d7113848-b248-4ab7-996b-504e674a3e6f/volumes"
Nov 01 21:40:23.364623 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:40:23.364540    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:40:23.737948 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:23.737917    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"]
Nov 01 21:40:23.738381 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:23.738105    4872 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" podUID="bf399806-7e4a-47eb-adf7-9fc43502e47b" containerName="maas-api" containerID="cri-o://2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d" gracePeriod=30
Nov 01 21:40:23.986891 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:23.986868    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:40:24.069030 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.068974    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk\" (UniqueName: \"kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk\") pod \"bf399806-7e4a-47eb-adf7-9fc43502e47b\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") "
Nov 01 21:40:24.069132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.069070    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca\") pod \"bf399806-7e4a-47eb-adf7-9fc43502e47b\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") "
Nov 01 21:40:24.069132 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.069110    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs\") pod \"bf399806-7e4a-47eb-adf7-9fc43502e47b\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") "
Nov 01 21:40:24.069216 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.069193    4872 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\") pod \"bf399806-7e4a-47eb-adf7-9fc43502e47b\" (UID: \"bf399806-7e4a-47eb-adf7-9fc43502e47b\") "
Nov 01 21:40:24.069495 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.069473    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca" pod "bf399806-7e4a-47eb-adf7-9fc43502e47b" (UID: "bf399806-7e4a-47eb-adf7-9fc43502e47b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:40:24.071178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.071155    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls" pod "bf399806-7e4a-47eb-adf7-9fc43502e47b" (UID: "bf399806-7e4a-47eb-adf7-9fc43502e47b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:40:24.071273 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.071240    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs" pod "bf399806-7e4a-47eb-adf7-9fc43502e47b" (UID: "bf399806-7e4a-47eb-adf7-9fc43502e47b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:40:24.071273 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.071238    4872 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk" pod "bf399806-7e4a-47eb-adf7-9fc43502e47b" (UID: "bf399806-7e4a-47eb-adf7-9fc43502e47b"). InnerVolumeSpecName "kube-api-access-nw5sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:40:24.170708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.170685    4872 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-maas-api-tls\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:24.170708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.170707    4872 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw5sk\" (UniqueName: \"kubernetes.io/projected/bf399806-7e4a-47eb-adf7-9fc43502e47b-kube-api-access-nw5sk\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:24.170708 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.170718    4872 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf399806-7e4a-47eb-adf7-9fc43502e47b-openshift-service-ca\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:24.170896 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.170726    4872 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf399806-7e4a-47eb-adf7-9fc43502e47b-metrics-certs\") on node \"ip-10-0-131-188.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:24.383298 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.383208    4872 generic.go:356] "Generic (PLEG): container finished" podID="bf399806-7e4a-47eb-adf7-9fc43502e47b" containerID="2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d" exitCode=0
Nov 01 21:40:24.383298 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.383267    4872 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"
Nov 01 21:40:24.383298 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.383287    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" event={"ID":"bf399806-7e4a-47eb-adf7-9fc43502e47b","Type":"ContainerDied","Data":"2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d"}
Nov 01 21:40:24.383550 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.383374    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq" event={"ID":"bf399806-7e4a-47eb-adf7-9fc43502e47b","Type":"ContainerDied","Data":"2e27f52a0bcc1521cb560f5b88a86f01869289fff577c17053cfcd1751129157"}
Nov 01 21:40:24.383550 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.383403    4872 scope.go:122] "RemoveContainer" containerID="2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d"
Nov 01 21:40:24.393580 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.393559    4872 scope.go:122] "RemoveContainer" containerID="2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d"
Nov 01 21:40:24.393880 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:40:24.393857    4872 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d\": container with ID starting with 2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d not found: ID does not exist" containerID="2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d"
Nov 01 21:40:24.393978 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.393885    4872 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d"} err="failed to get container status \"2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d\": rpc error: code = NotFound desc = could not find container \"2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d\": container with ID starting with 2c12f6966c3356a1c637a59eef4607dd2d29e34faa944e3be84b6f0c3f96da5d not found: ID does not exist"
Nov 01 21:40:24.547627 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.547602    4872 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"]
Nov 01 21:40:24.549951 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:24.549934    4872 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-e23c3b-68659d479b-5nmhq"]
Nov 01 21:40:25.077697 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:25.077610    4872 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf399806-7e4a-47eb-adf7-9fc43502e47b" path="/var/lib/kubelet/pods/bf399806-7e4a-47eb-adf7-9fc43502e47b/volumes"
Nov 01 21:40:43.733850 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:43.733823    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:40:43.739564 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:43.739542    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:40:43.740285 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:43.740266    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9m5fw_b245f314-7357-4797-90d0-aa95f77b154a/console-operator/2.log"
Nov 01 21:40:43.746119 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:40:43.746102    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9n_9e3961b5-f2f0-4c47-9017-72de63a2f7c4/ovn-acl-logging/0.log"
Nov 01 21:40:53.366681 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:40:53.366634    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:41:13.728803 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.728772    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"]
Nov 01 21:41:13.729186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729161    4872 cpu_manager.go:414] "containerMap: removing container" podUID="d7113848-b248-4ab7-996b-504e674a3e6f" containerName="payload-processing"
Nov 01 21:41:13.729186 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729172    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7113848-b248-4ab7-996b-504e674a3e6f" containerName="payload-processing"
Nov 01 21:41:13.729288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729190    4872 cpu_manager.go:414] "containerMap: removing container" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy"
Nov 01 21:41:13.729288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729195    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy"
Nov 01 21:41:13.729288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729224    4872 cpu_manager.go:414] "containerMap: removing container" podUID="bf399806-7e4a-47eb-adf7-9fc43502e47b" containerName="maas-api"
Nov 01 21:41:13.729288 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729230    4872 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf399806-7e4a-47eb-adf7-9fc43502e47b" containerName="maas-api"
Nov 01 21:41:13.729416 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729305    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="2637e83c-ebe9-41ba-89b1-10d57a44ff6d" containerName="istio-proxy"
Nov 01 21:41:13.729416 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729318    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf399806-7e4a-47eb-adf7-9fc43502e47b" containerName="maas-api"
Nov 01 21:41:13.729416 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.729327    4872 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7113848-b248-4ab7-996b-504e674a3e6f" containerName="payload-processing"
Nov 01 21:41:13.732523 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.732507    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.813550 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813509    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.813791 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813554    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.813791 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813742    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cd8q\" (UniqueName: \"kubernetes.io/projected/2bbc69e6-9f85-4936-9c7b-948998029de3-kube-api-access-7cd8q\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.813928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813855    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.813928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813898    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbc69e6-9f85-4936-9c7b-948998029de3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.813928 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813924    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.814079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.813949    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.814079 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.814043    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.814179 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.814081    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915018 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.914971    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915229 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915034    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915229 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915061    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915229 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915110    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cd8q\" (UniqueName: \"kubernetes.io/projected/2bbc69e6-9f85-4936-9c7b-948998029de3-kube-api-access-7cd8q\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915229 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915172    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915229 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915218    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbc69e6-9f85-4936-9c7b-948998029de3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915253    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915284    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915322    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.915506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915418    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.916053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.915993    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbc69e6-9f85-4936-9c7b-948998029de3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.916191 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.916051    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.916191 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.916091    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.916191 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.916139    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.917500 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.917476    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.918299 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.918276    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.936915 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.936880    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"]
Nov 01 21:41:13.972881 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.972849    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbc69e6-9f85-4936-9c7b-948998029de3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:13.973056 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:13.973031    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cd8q\" (UniqueName: \"kubernetes.io/projected/2bbc69e6-9f85-4936-9c7b-948998029de3-kube-api-access-7cd8q\") pod \"maas-default-gateway-openshift-default-58c6585447-5t72t\" (UID: \"2bbc69e6-9f85-4936-9c7b-948998029de3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:14.045463 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.045362    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:14.479814 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.479789    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"]
Nov 01 21:41:14.482348 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:41:14.482316    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbc69e6_9f85_4936_9c7b_948998029de3.slice/crio-68c08352c6dcd073c637c658fad1656dd17e5d9360b6c4d8b08696aa658afec3 WatchSource:0}: Error finding container 68c08352c6dcd073c637c658fad1656dd17e5d9360b6c4d8b08696aa658afec3: Status 404 returned error can't find the container with id 68c08352c6dcd073c637c658fad1656dd17e5d9360b6c4d8b08696aa658afec3
Nov 01 21:41:14.484880 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.484839    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:41:14.484958 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.484930    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:41:14.484995 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.484979    4872 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:41:14.633335 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.632720    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" event={"ID":"2bbc69e6-9f85-4936-9c7b-948998029de3","Type":"ContainerStarted","Data":"dac4d7e61d08f5e2cb800b59334e4df8df6f1098b8dea987ef2a423148a04e01"}
Nov 01 21:41:14.633335 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.632786    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" event={"ID":"2bbc69e6-9f85-4936-9c7b-948998029de3","Type":"ContainerStarted","Data":"68c08352c6dcd073c637c658fad1656dd17e5d9360b6c4d8b08696aa658afec3"}
Nov 01 21:41:14.751497 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:14.751375    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podStartSLOduration=1.7513607690000002 podStartE2EDuration="1.751360769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:14.74906993 +0000 UTC m=+4231.798179772" watchObservedRunningTime="2026-11-01 21:41:14.751360769 +0000 UTC m=+4231.800470586"
Nov 01 21:41:15.049739 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:15.049058    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:15.055363 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:15.055015    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:15.055363 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:15.055077    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:16.046302 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:16.046250    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:16.047026 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:16.046977    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:17.045886 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:17.045848    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:17.046082 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:17.045917    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:18.045917 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:18.045881    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:18.046417 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:18.045942    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:19.045829 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:19.045791    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:19.046020 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:19.045858    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:20.045755 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:20.045717    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:20.045931 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:20.045773    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:21.046543 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:21.046493    4872 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5t72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 21:41:21.046972 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:21.046565    4872 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t" podUID="2bbc69e6-9f85-4936-9c7b-948998029de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 21:41:22.788984 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:22.788950    4872 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:22.789516 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:22.789418    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:22.790398 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:22.790378    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5t72t"
Nov 01 21:41:23.366803 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:41:23.366760    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:41:44.311978 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:44.311945    4872 scope.go:122] "RemoveContainer" containerID="09b48e7b3bb90d38baf1e29abcff6d9cc150fc9696205a7d3078796c36968dea"
Nov 01 21:41:48.885678 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:48.885632    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f499969f4-rlkg6_e7f0665b-f370-478a-a7d2-abcc6021bfa8/manager/0.log"
Nov 01 21:41:50.407497 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:50.407468    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4778f-bt8rf_c54edc37-7504-492c-8386-5bfef499d4fa/manager/0.log"
Nov 01 21:41:51.034326 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:51.034297    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-plqqn_e597514c-f313-41a8-975e-06872e130081/manager/0.log"
Nov 01 21:41:51.291940 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:51.291868    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tkz8s_f4641e48-f529-4403-b5ab-cb9c7537970e/proxy/0.log"
Nov 01 21:41:52.251289 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:52.251263    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fb7d7759-5962m_02f9e85d-56dc-4207-be48-b8b68211cf17/maas-api/0.log"
Nov 01 21:41:53.364414 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:41:53.364382    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:41:58.137053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:58.136966    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b9bf4598-hm78q_aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc/kube-auth-proxy/0.log"
Nov 01 21:41:58.774806 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:58.774768    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-5t72t_2bbc69e6-9f85-4936-9c7b-948998029de3/istio-proxy/0.log"
Nov 01 21:41:59.943476 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:41:59.943447    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-nd97z_4102344f-e1f1-4c5a-a1b9-a57ba4e86b30/istio-proxy/0.log"
Nov 01 21:42:00.210624 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:00.210547    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-qnwxz_a11b393c-7ffa-4964-b8bb-1962b54592e5/istio-proxy/0.log"
Nov 01 21:42:01.116878 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:01.116853    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-jdsm2_fd95e301-fe56-462e-9a29-f83709bb22b5/payload-pre-processing/0.log"
Nov 01 21:42:01.335758 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:01.335727    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mp5sx_fa5c0b0a-df95-4568-a3b5-ffdb6ace8352/payload-processing/0.log"
Nov 01 21:42:01.555088 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:01.555059    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ff46cf4c-rt6qk_b20d35a5-f97b-48a6-9dfb-73c6d656032c/router/0.log"
Nov 01 21:42:02.642902 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:02.642872    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-t4dq6_7cfc3889-952c-448a-98ef-8098113fa769/main/0.log"
Nov 01 21:42:02.850053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:02.850028    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rgff6_5d760d74-afb7-4acf-896c-7afc9099a392/main/0.log"
Nov 01 21:42:03.065684 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:03.065635    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-86g27_f3a653fd-4a69-48ab-a68b-ffc0011441cf/main/0.log"
Nov 01 21:42:03.280300 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:03.280266    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv_d99c741f-b128-4269-81e1-af775684f2c0/main/0.log"
Nov 01 21:42:03.497834 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:03.497806    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6xbbn_1920fad9-b098-4cd7-8786-62261b1bb34b/main/0.log"
Nov 01 21:42:03.920434 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:03.920407    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm_e2403ce9-732c-42f0-b2d4-eb1fb462e1d6/main/0.log"
Nov 01 21:42:13.441455 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:13.441420    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f499969f4-rlkg6_e7f0665b-f370-478a-a7d2-abcc6021bfa8/manager/0.log"
Nov 01 21:42:14.749311 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:14.749281    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4778f-bt8rf_c54edc37-7504-492c-8386-5bfef499d4fa/manager/0.log"
Nov 01 21:42:15.274832 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:15.274805    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-plqqn_e597514c-f313-41a8-975e-06872e130081/manager/0.log"
Nov 01 21:42:15.538214 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:15.538137    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tkz8s_f4641e48-f529-4403-b5ab-cb9c7537970e/proxy/0.log"
Nov 01 21:42:16.042535 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:16.042506    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fb7d7759-5962m_02f9e85d-56dc-4207-be48-b8b68211cf17/maas-api/0.log"
Nov 01 21:42:21.279764 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:21.279738    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b9bf4598-hm78q_aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc/kube-auth-proxy/0.log"
Nov 01 21:42:21.929514 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:21.929484    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-5t72t_2bbc69e6-9f85-4936-9c7b-948998029de3/istio-proxy/0.log"
Nov 01 21:42:23.089413 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:23.089383    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-nd97z_4102344f-e1f1-4c5a-a1b9-a57ba4e86b30/istio-proxy/0.log"
Nov 01 21:42:23.354042 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:23.353952    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-qnwxz_a11b393c-7ffa-4964-b8bb-1962b54592e5/istio-proxy/0.log"
Nov 01 21:42:23.364281 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:42:23.364252    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:42:24.226525 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:24.226494    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-jdsm2_fd95e301-fe56-462e-9a29-f83709bb22b5/payload-pre-processing/0.log"
Nov 01 21:42:24.434037 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:24.434008    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mp5sx_fa5c0b0a-df95-4568-a3b5-ffdb6ace8352/payload-processing/0.log"
Nov 01 21:42:24.641492 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:24.641422    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ff46cf4c-rt6qk_b20d35a5-f97b-48a6-9dfb-73c6d656032c/router/0.log"
Nov 01 21:42:25.474421 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:25.474389    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-t4dq6_7cfc3889-952c-448a-98ef-8098113fa769/main/0.log"
Nov 01 21:42:25.680203 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:25.680172    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rgff6_5d760d74-afb7-4acf-896c-7afc9099a392/main/0.log"
Nov 01 21:42:25.896076 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:25.896052    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-86g27_f3a653fd-4a69-48ab-a68b-ffc0011441cf/main/0.log"
Nov 01 21:42:26.101922 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:26.101895    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87wvlv_d99c741f-b128-4269-81e1-af775684f2c0/main/0.log"
Nov 01 21:42:26.309161 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:26.309086    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6xbbn_1920fad9-b098-4cd7-8786-62261b1bb34b/main/0.log"
Nov 01 21:42:26.744849 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:26.744820    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5nzlm_e2403ce9-732c-42f0-b2d4-eb1fb462e1d6/main/0.log"
Nov 01 21:42:52.807303 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:52.807264    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4778f-bt8rf_c54edc37-7504-492c-8386-5bfef499d4fa/manager/0.log"
Nov 01 21:42:53.362313 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:53.362274    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4778f-bt8rf_c54edc37-7504-492c-8386-5bfef499d4fa/manager/0.log"
Nov 01 21:42:53.375687 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:42:53.375628    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:42:54.095960 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:54.095933    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mp5sx_fa5c0b0a-df95-4568-a3b5-ffdb6ace8352/payload-processing/0.log"
Nov 01 21:42:54.309867 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:42:54.309837    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mp5sx_fa5c0b0a-df95-4568-a3b5-ffdb6ace8352/payload-processing/0.log"
Nov 01 21:43:23.364019 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:43:23.363979    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:43:28.394895 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:28.394811    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4778f-bt8rf_c54edc37-7504-492c-8386-5bfef499d4fa/manager/0.log"
Nov 01 21:43:28.952764 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:28.952730    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4778f-bt8rf_c54edc37-7504-492c-8386-5bfef499d4fa/manager/0.log"
Nov 01 21:43:29.661938 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:29.661906    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mp5sx_fa5c0b0a-df95-4568-a3b5-ffdb6ace8352/payload-processing/0.log"
Nov 01 21:43:29.880441 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:29.880413    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mp5sx_fa5c0b0a-df95-4568-a3b5-ffdb6ace8352/payload-processing/0.log"
Nov 01 21:43:30.213575 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.213543    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r24pw/must-gather-ssg6g"]
Nov 01 21:43:30.217310 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.217290    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.295125 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.295092    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsl27\" (UniqueName: \"kubernetes.io/projected/e0d10246-039c-4ca9-8115-8f7b76a7a915-kube-api-access-wsl27\") pod \"must-gather-ssg6g\" (UID: \"e0d10246-039c-4ca9-8115-8f7b76a7a915\") " pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.295296 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.295141    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0d10246-039c-4ca9-8115-8f7b76a7a915-must-gather-output\") pod \"must-gather-ssg6g\" (UID: \"e0d10246-039c-4ca9-8115-8f7b76a7a915\") " pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.321053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.321018    4872 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r24pw\"/\"default-dockercfg-pp4xf\""
Nov 01 21:43:30.321053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.321038    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r24pw\"/\"openshift-service-ca.crt\""
Nov 01 21:43:30.321053 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.321017    4872 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r24pw\"/\"kube-root-ca.crt\""
Nov 01 21:43:30.322853 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.322830    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/must-gather-ssg6g"]
Nov 01 21:43:30.396130 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.396085    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsl27\" (UniqueName: \"kubernetes.io/projected/e0d10246-039c-4ca9-8115-8f7b76a7a915-kube-api-access-wsl27\") pod \"must-gather-ssg6g\" (UID: \"e0d10246-039c-4ca9-8115-8f7b76a7a915\") " pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.396327 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.396148    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0d10246-039c-4ca9-8115-8f7b76a7a915-must-gather-output\") pod \"must-gather-ssg6g\" (UID: \"e0d10246-039c-4ca9-8115-8f7b76a7a915\") " pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.396525 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.396501    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0d10246-039c-4ca9-8115-8f7b76a7a915-must-gather-output\") pod \"must-gather-ssg6g\" (UID: \"e0d10246-039c-4ca9-8115-8f7b76a7a915\") " pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.454317 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.454287    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsl27\" (UniqueName: \"kubernetes.io/projected/e0d10246-039c-4ca9-8115-8f7b76a7a915-kube-api-access-wsl27\") pod \"must-gather-ssg6g\" (UID: \"e0d10246-039c-4ca9-8115-8f7b76a7a915\") " pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.531439 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.531356    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/must-gather-ssg6g"
Nov 01 21:43:30.901596 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.901566    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/must-gather-ssg6g"]
Nov 01 21:43:30.904379 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:43:30.904349    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0d10246_039c_4ca9_8115_8f7b76a7a915.slice/crio-52fc13141adf03bf6b711ad76cc4b1e68a58af1e43a453e11c8e64d840446609 WatchSource:0}: Error finding container 52fc13141adf03bf6b711ad76cc4b1e68a58af1e43a453e11c8e64d840446609: Status 404 returned error can't find the container with id 52fc13141adf03bf6b711ad76cc4b1e68a58af1e43a453e11c8e64d840446609
Nov 01 21:43:30.906178 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:30.906159    4872 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:43:31.290258 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:31.290174    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/must-gather-ssg6g" event={"ID":"e0d10246-039c-4ca9-8115-8f7b76a7a915","Type":"ContainerStarted","Data":"52fc13141adf03bf6b711ad76cc4b1e68a58af1e43a453e11c8e64d840446609"}
Nov 01 21:43:33.302613 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:33.302571    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/must-gather-ssg6g" event={"ID":"e0d10246-039c-4ca9-8115-8f7b76a7a915","Type":"ContainerStarted","Data":"0ebc9d600cb3441e4daad680f06e4e5f922ce925bd575bab22d4762068da1933"}
Nov 01 21:43:33.303137 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:33.302627    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/must-gather-ssg6g" event={"ID":"e0d10246-039c-4ca9-8115-8f7b76a7a915","Type":"ContainerStarted","Data":"8c5e4249c90302d2a402445c791f6bac99c5289ddb015e955518d3f71709cdf4"}
Nov 01 21:43:33.416881 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:33.416803    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-r24pw/must-gather-ssg6g" podStartSLOduration=1.878927249 podStartE2EDuration="3.416784486s" totalImagesPullingTime="1.537857237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:33.416478608 +0000 UTC m=+4370.465588429" watchObservedRunningTime="2026-11-01 21:43:33.416784486 +0000 UTC m=+4370.465894305"
Nov 01 21:43:39.996442 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:39.996410    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8x2dv_1cdfd004-c495-4f69-9aba-35276b089b06/global-pull-secret-syncer/0.log"
Nov 01 21:43:41.210604 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:41.210567    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-75vgh_bd018011-2b75-4090-812a-5b5039164aa9/connection-checker/0.log"
Nov 01 21:43:41.651674 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:41.648246    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-76pfm_9b44fcb4-4d02-4b52-b0bb-b67dcaafd336/connection-checker/0.log"
Nov 01 21:43:42.088366 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:42.088284    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pjl9s_ace44fda-7418-4639-83db-e7499a8e140d/connection-checker/0.log"
Nov 01 21:43:43.504189 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:43.504130    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f7pr8_32264030-2487-4a8d-b5e4-3f233831760b/konnectivity-agent/0.log"
Nov 01 21:43:43.913097 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:43.913051    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-188.us-west-1.compute.internal_434dabbcb9fcd49f00b858143ea6ae44/haproxy/0.log"
Nov 01 21:43:49.724845 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.724813    4872 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"]
Nov 01 21:43:49.732575 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.732540    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:49.785943 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.785910    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"]
Nov 01 21:43:49.900466 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.900432    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-lib-modules\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:49.900640 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.900485    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmsh\" (UniqueName: \"kubernetes.io/projected/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-kube-api-access-ftmsh\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:49.900640 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.900572    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-podres\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:49.900766 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.900713    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-proc\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:49.900766 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:49.900739    4872 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-sys\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.002813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002712    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-lib-modules\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.002813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002773    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmsh\" (UniqueName: \"kubernetes.io/projected/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-kube-api-access-ftmsh\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.002813 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002795    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-podres\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.003112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002865    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-proc\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.003112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002881    4872 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-sys\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.003112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002894    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-lib-modules\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.003112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.002982    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-podres\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.003112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.003038    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-proc\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.003112 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.003060    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-sys\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.060212 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.060185    4872 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmsh\" (UniqueName: \"kubernetes.io/projected/789cea5c-47a0-41d8-a2d8-d28c89d6f95b-kube-api-access-ftmsh\") pod \"perf-node-gather-daemonset-4fdnc\" (UID: \"789cea5c-47a0-41d8-a2d8-d28c89d6f95b\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.346022 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.345951    4872 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:50.720411 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:50.720382    4872 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"]
Nov 01 21:43:50.723491 ip-10-0-131-188 kubenswrapper[4872]: W1101 21:43:50.723462    4872 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod789cea5c_47a0_41d8_a2d8_d28c89d6f95b.slice/crio-0a04df33479c814befa05a6477459fbe9df5c8c5267f7cb79f375a8e463d0c91 WatchSource:0}: Error finding container 0a04df33479c814befa05a6477459fbe9df5c8c5267f7cb79f375a8e463d0c91: Status 404 returned error can't find the container with id 0a04df33479c814befa05a6477459fbe9df5c8c5267f7cb79f375a8e463d0c91
Nov 01 21:43:51.444768 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:51.444082    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc" event={"ID":"789cea5c-47a0-41d8-a2d8-d28c89d6f95b","Type":"ContainerStarted","Data":"55d0642b90e81428f6ef4b9b717032ada070e64dd1e7b36b788055b62aae062f"}
Nov 01 21:43:51.444768 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:51.444148    4872 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc" event={"ID":"789cea5c-47a0-41d8-a2d8-d28c89d6f95b","Type":"ContainerStarted","Data":"0a04df33479c814befa05a6477459fbe9df5c8c5267f7cb79f375a8e463d0c91"}
Nov 01 21:43:51.444768 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:51.444732    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:51.558934 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:51.558878    4872 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc" podStartSLOduration=2.558863561 podStartE2EDuration="2.558863561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:51.557674043 +0000 UTC m=+4388.606783857" watchObservedRunningTime="2026-11-01 21:43:51.558863561 +0000 UTC m=+4388.607973379"
Nov 01 21:43:53.365123 ip-10-0-131-188 kubenswrapper[4872]: E1101 21:43:53.365087    4872 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ntncn" containerName="extractor"
Nov 01 21:43:57.299681 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:57.299627    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-7rw9k_83552290-2377-4a32-be4c-e1f10c121b12/cluster-monitoring-operator/0.log"
Nov 01 21:43:57.459365 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:57.459335    4872 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-4fdnc"
Nov 01 21:43:59.876506 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:43:59.876476    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gmd9_c2f1e2ba-c96e-4185-a335-c8f4e44ec422/node-exporter/0.log"
Nov 01 21:44:00.291500 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:44:00.291469    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gmd9_c2f1e2ba-c96e-4185-a335-c8f4e44ec422/kube-rbac-proxy/0.log"
Nov 01 21:44:00.706833 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:44:00.706806    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gmd9_c2f1e2ba-c96e-4185-a335-c8f4e44ec422/init-textfile/0.log"
Nov 01 21:44:13.210029 ip-10-0-131-188 kubenswrapper[4872]: I1101 21:44:13.209998    4872 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b9bf4598-hm78q_aa4e14e6-d5df-4f4a-8cd0-f876eb22cbbc/kube-auth-proxy/0.log"